When an app that requires network connectivity is sent to Apple for a review, watching them review the app is straightforward. You just need to take a peek at the server logs.
A couple months ago, Apple used to be very fast in order to approve new apps and new versions of existing apps. Apparently, it’s no longer the case and it can easily take a solid 3 weeks before your app can eventually hit the store.
What exactly does happen during the “in review” step?
After the initial submission of one of our app had been switched to the “in review” state, there was no sign of any connections during 2 weeks.
Then somebody finally logged in.
We provided a test account, and the Apple guy did use it. He apparently initially made a typo while entering the login and/or password.
This suggests that reviewers are using real devices, not emulators. The user agent described a 3GS device with an up-to-date iOS. The push service was enabled.
Since this is a location-based app, it was also easy to look at where the reviewer connected from. The GPS clearly stated that he was working from Cupertino, and the fact that the position was constantly changing confirms that reviewers are probably using real devices and not emulators. The IP address he connected from was also one of Apple’s network.
A common belief is that Apple may delegate the reviewing process to third-party companies. But it apparently turns out to be false: reviewers look like in-house employees.
The reviewer left the app after 1 minute and no actions.
Four days later, another reviewer (or possibly the same one, but IP addresses were different) connected. This time, he stayed online for like 5 minutes. During this time, he went further than just browing screens, and actually published content. While 5 minutes can look pretty short, this is probably enough to immediately spot a crashy app or an app that doesn’t match its description.
The app remained in the “in review” state for like 10 days until it finally hit the store. There was no other footprints of a connection in-between.
So what happened during the “in review” state besides the 5 minutes a reviewer actually used the app remains a mystery.
Then came the updates. Two versions followed the initial one.
The review process also took quite a while (2+ weeks). While the app status quickly switched to “in review”, nothing happened server-side during 10 days.
Finally, somebody connected to the test account. And immediately logged out. Two days later, a new reviewer connected (yet from another IP address), quickly cruised around a couple screens and logged out. The app was pushed to the AppStore on the same day.
Third version: as always, it turned from “waiting for review” to the “in review” state in 3 days. It stuck to this state for 8 days. But no one ever used the test account during this period. No matter what, it was pushed to the store.
Apparently, there’s less and less application testing accross different versions of the same app.
But what might happen during the “in review” process then? Are they analyzing the code?
Apple is working hard at trying to avoid malicious applications (and from a user perspective, this is a great advantage of the AppStore over open markets, albeit not bullet-proof). Do they actually spend all the reviewing process at manually reverse-engineering the code? I can hardly imagine they actually do that. They run static analyzers for sure, but they probably requires a few milliseconds to run.
So why does the “in review” state last for so long?