On App Store Reviews

Monday, 2008-08-11; 15:04:26

reviews from those who haven't used an app are still useful

Some iPhone developers in the community have called for iTunes App Store reviews to be restricted to only those who have actually used the app on their iPhone and iPod touch. The proposal is that Apple should track whether someone has actually downloaded an app, and then only allow users who have done so to review that app.

I think this is a terrible idea. The most fundamental reason why relates to statistics. But let’s first go through some of the less convincing arguments.

Overly-Broad Generalization

There are two assumptions underlying the argument that reviews should be limited to those who downloaded the app. The first is that those who downloaded the app to their iPhone are the only ones who have used the app.

But with the iPhone getting so much hype, it’s hard to live anymore without knowing someone who has an iPhone. Personally, I have two family members who have an iPhone, and my roommate has an iPhone as well. I’ve been reading articles about various iPhone apps, and have recommended many to my brother based on reviews I’ve read, despite the fact that I don’t have an iPhone myself.

The reason I’ve done so is not only to get help my brother find apps he wants, but to be able to try out those apps myself when my brother’s not using his iPhone. I’ve personally played several free games on the App Store such as Tap Tap Revenge, Spinner, CubeRunner, etc. I’ve also used one app that my brother had to pay for: Tiny Violin.

I have valid criticisms that I can make about all of these apps. When playing TTR with shakes, I have a huge tendency to accidentally touch the screen and pause the game when I’m shaking to the left or right. It annoys me to no end! One time, Spinner started erroneously reading the accelerometer and the playing field shook ridiculously out of control, even though other accelerometer-based games worked fine. CubeRunner gets way too tedious at times, and the author of Tiny Violin has the super-annoying tendency to choose violin samples that aren’t sad enough for the intended use of the program. (What’s the point of feigning sympathy for another person when the song that Tiny Violin plays is upbeat?)

Are people seriously contending that I should be banned from writing reviews about these apps that I’ve used? I can download the apps to my computer even though I don’t have an iPhone or iPod touch, and then review the apps, but why should I have to go to that trouble (or even pay the extra money) to do so?

Useful Criticisms

The second assumption is that people cannot make valid criticisms about an app without having used it. This may be a controversial opinion, but I think that this assumption is treading on very shaky ground as well.

The Mac community, in particular, is very obsessed about UI. The UI for a Mac program can be a real put-off to many Mac users, whether or not the underlying functionality is useful or not. Just take the example of TripLog/1040, an app whose interface is so horrible that it was widely maligned on Mac-related weblogs even before it was even released. This is more likely to happen in the Mac community—and by extension the iPhone community—because we’re so particular about our UIs.

But a UI criticism on a Mac-oriented website is just as much a review as an official review in the iTunes Store. And yet, some are saying that it’s OK to publish the former without using the app, but it’s not OK to do the latter? How is it reasonable for reviews on iTunes to be restricted to users who have used the app?

A review on a weblog is just as powerful as a review on iTunes, if not more so. Personally, I rarely look at reviews on iTunes to decide whether or not to purchase a song on iTunes, or to download an app from the web. On iTunes, I’ll listen to the 30-second sample myself, and for apps, I’ll look at the screenshots and the description of the functionality, and test out a time-limited trial if one is available.

But when I see a mockery of an app on Daring Fireball (even if it’s a one-line zinger), I’m more apt to notice. I don’t think this is very rare, either. A large portion of the Mac community has a very personal stake in their computer platform, and a large portion of them probably subscribe to Daring Fireball alone, not to mention the multitude of other websites that picked up on the mockery, and the reverberating conversations through Twitter. The criticisms of the TripLog UI caused the developer himself to respond to the criticisms, as he probably perceived that the press he was getting was going to dampen his sales. Would a single review on iTunes have had as much of an effect?

At the same time, were the criticisms of the UI not valid? I’d argue that they were pretty spot on, as I agree that the UI looks horrible. So if UI is important, and the UI is horrible, isn’t that an argument to not buy the app? Isn’t that a valid review? Why not allow the same things on iTunes?

Statistical Bias

Both of the two preceding arguments pale in comparison to this one: disallowing reviews from non-buyers artificially inflates App Store ratings. The tendency of these reviews to be high is so strong that it would make the iTunes reviews completely useless.

Consider the fact that most paid apps on the App Store don’t have time-limited or feature-limited demos that allow users to try out the app before paying for it. There are some exceptions, such as Twinkle Exposure and Twiterrific, but these demos usually don’t exist on the iPhone.

Suppose there are two iPhone users, user U who thinks app A for $5 is worth it, and user V who thinks app A for $5 is not worth it. U purchases and downloads app A, and gives it a five-star rating. V does not purchase app A, and is disallowed from reviewing app A. App A retains its five-star rating, despite the fact that only 50% of the users thought the app was worth buying.

Furthermore, consider user W who is on the fence about purchasing app A. Suppose he gives it the benefit of the doubt, and decides to purchase and download app A. After doing so, there is a high psychological tendency for user W to rationalize his purchase so that he doesn’t feel like he’s wasted his money. Even if user W sees faults in app A, the tendency is to give more weight to the good aspects of the app and less weight to the bad aspects. The effect is a further inflation of app A’s average rating.

Restricting reviews to buyers would simply cause ratings to not reflect the actual value of the app.


The underlying point is that price is absolutely a criterion on which to rate an app. Reviews exist precisely to guide other potential customers about whether an app is worth it at a certain price. If the price goes down, more users will ultimately deem the app worth it at that price.

That non-buyers will post negative reviews about an app is absolutely useful. It means that some people think that the features an app offers is not enough to justify paying a certain amount of money for that app. If I tell someone that a Prius is not worth $100,000, should Toyota be allowed to intervene and say, “Nope, sorry, you haven’t purchased a Prius, so your opinion is invalid.” I hope everyone agrees that this is ludicrous. Same thing goes for the App Store.

If a developer thinks that his app is worth $50, and users are only willing to pay $5 and they register their views on iTunes, tough fucking luck. The price people are willing to pay is not based on the price you set, but on the price people are willing to pay. Customers are not paying you for the time you put into the app; they’re paying you for how useful it is to them.

If you want to participate in a capitalistic system (whether it’s the iTunes App Store or the larger societal structure of some nations), be prepared to accept the consequences: some people don’t think that your work is worth it. There may be whiners who don’t want to pay for anything—some of which are open source whiners—but bitching about some consumers not wanting to purchase your app at a specific price is even more whiny.

App Store Shareware

Currently, the App Store effectively prohibits shareware from being listed. The current method for allowing users to try out an app is not shareware, it’s adware; that is, there is a free version of the app that includes occasional advertisements, and a paid version of the app that removes the ads. Feature-limited demos are possible on the App Store, but time-limited shareware demos are not.

Time-limited shareware demos on the Mac work like this: you download a single program, and it allows 30 days of free usage. When you’re ready to buy, you go to the website and purchase the product, receive a serial number, and then enter it into the app that you’ve already-downloaded.

Shareware could still work on the App Store with the current system, but it would be a little more complicated: there’d be a time-limited free app, and a separate app that you pay for. You’d have to download the (essentially) same app again when you pay for it.

The problem is that the data for each app on the App Store is sandboxed when you install it on an iPhone and iPod touch. So when you finally download and pay for the app, you’re actually installing a “new” app, and you’d lose all your data from the time-limited demo, unless they both offered internet syncing.

Not only that, but when a user deletes an app from his iPhone or iPod touch, all related preference and data files are deleted as well (as far as I know). So there’d be no way to enforce the time-limitation: the user could delete the app and then re-download it, and he’d have another portion of time to use the app. If the app used internet syncing to mitigate the first problem, then it’d exacerbate this second problem because a user wouldn’t be fearful of losing his data after deleting the time-limited app. (It’s possible that on first run, the app would obtain a unique ID and prevent syncing from a previous time-trial session, and therefore discourage this practice by encouraging data loss between time-trial uses. But this seems both overly-complicated and potentially dangerous.)

The problem with the rating system—that people can’t try out apps before they buy—is exacerbated by the lack of a way to put shareware on the App Store. On the Mac, many programs are shareware, and allow people to actually get valuable insight on the use of an app before being forced to pay for it to continue using it. This is not the case on the App Store.

So in the App Store’s case, negative reviews are more skewed toward users who haven’t downloaded an app. But that doesn’t mean those reviews aren’t useful.

And even if the App Store did have a mechanism for allowing shareware to be listed, the problem still wouldn’t go away completely. Independent Mac developers tend to use the shareware model, but it doesn’t guarantee that there won’t be apps that only offer a paid version. Thus, the afore-explained argument of statistical bias will still apply even if the App Store was less restrictive. This inevitable situation cannot be escaped.

A Real Proposal

Now, let’s not go overboard here. Reviews from those who have not used an app can be just as irrational and overly negative as those who have used an app and have had personal experience with it. And I agree that those who have used an app probably have a more reasonable view of the utility of an app. But reviews from users who haven’t downloaded an app are still useful.

So here’s how I think iTunes Store reviews should be changed:

  1. Individual reviews from users who have not downloaded an app should be marked as such with a badge.
  2. Average ratings should be calculated in a weighted manner: ratings from those who’ve downloaded an app should influence the average rating more than those who haven’t downloaded that same app.

Details, such as what the badge says, what it looks like, and the particulars of the weighting system are left to readers, or more likely, to Apple.

Technological Supernova   Rants   Older   Newer   Post a Comment