Sunday, February 12, 2012

Why Ratings and Reviews Suck and How to Save Them

The use of consumer ratings and reviews has already been a game-changer online. Ratings and Reviews (R&Rs) were the first true use of social media to support and enhance commerce, predating ads on Facebook, Groupon discounts and viral videos on YouTube. While some of us participated on a BBS or Usenet Board in the early days of the Internet, the first time most people posted anything for strangers to read was not a status update on MySpace but a R&R on Amazon, TripAdvisor or Yelp.

It is the success and maturity of R&Rs that leave me perplexed at how little innovation we have seen in the space and how much more the leaders could accomplish. This was the topic of conversation recently when I chatted with Jeremy Epstein, the new VP of Marketing at Sprinklr, a social media management service provider that Altimeter put on the top of its list of vendors to support large corporations. We both agreed that it is high time for the primary providers of online R&Rs to lead once again with new features that make R&Rs more powerful and reliable.

First, here is what is right about R&Rs: They remain the most trusted form of communication, lagging only recommendations from families and friends. According to Forrester (subscription required), consumer product R&Rs are trusted by 62% of US adults, compared to 57% who trust "expert ratings and reviews," 30% who trust company web sites and 23% who trust what they see on TV.

Here is what is wrong with R&Rs: That high level of trust may be about to evaporate. While I have no data, I sense more people having doubts as to the validity of the R&Rs they read online. These doubts may be well founded; recent articles have raised awareness of dubious or outright deceptive practices such as offering free product in exchange for ratings, people anonymously rating their employers' products, and farms of workers hired to post ratings.

If you are an authentic marketer and really want to get your blood boiling, check out, where people compete to write paid reviews. One person promises "I will leave 3 Amazon reviews from unique accounts that YOU write for $5," and 44 out of 45 reviews are positive. Comments include "I of course do always need more reviews, how many accounts do you have? And can you also do likes?" Obviously reputable brands are not on bidding for fake reviews, but the transparent sleaziness goes a long way to demonstrating the problem with R&Rs.

While those involved are smart enough to avoid saying, "We get paid to do positive ratings," the implication is loud and clear. "I would have done 4 stars instead of 5 without the deal," says one guy who received free product in exchange for his rating. And an employee of one of the R&R-writing services reports, "We were not asked to provide a five-star review, but would be asked to turn down an assignment if we could not give one."

In theory, it is not unethical to give discounts or freebies in exchange for a rating and review, provided a brand does not reward the effusiveness of the rating and--here is the part everyone misses--the consumer discloses the material relationship as required by FTC guidelines. (And just to be clear, it is the brand's responsibility not merely to instruct the consumer to reveal the arrangement but also to monitor that consumers are, in fact, disclosing appropriately.)

While I hope marketers are following the rules, I do not think the purveyors of R&Rs should rely on the willingness of those involved to adhere to rules--the stakes are just too high. For consumers, the temptation is great to earn cash, freebies and discounts in exchange for five minutes of faux exuberance on Yelp or Amazon. The reasons marketers are tempted to engage in dubious R&R practices are also obvious, as one recent study validated what we all intuitively believe about the value of positive ratings--each additional star on Yelp is worth 5% to 9% in incremental revenue.

Perhaps you believe you are a keen observer who can easily separate the fake reviews from the real ones. Turns out researchers have found the average consumer only gets this right around 50% of the time. You can even test yourself with an online quiz on the Marketplace site. (I scored 75% in the simple four-question quiz.) Even if we believe people have the power to spot and discount fake R&Rs, this cannot be the solution--it takes too long to review and assess individual reviews, and given the power of "star ratings" to furnish info at a glance and permit sorting based on score, we clearly need better ways to improve the quality and reliability of R&Rs.

Given the incentives to cheat, why haven't we seen innovations that encourage appropriate reviews and that filter reviews to make them more accurate and believable? A great deal of effort is going into technological advances to suss out fake reviews, but aren't there easier and better ways to accomplish this same goal?

For example, we are each unique, so why do we see the same information on most R&R sites? When I go to the San Antonio Things to Do page on TripAdvisor, the top-rated attraction in the city of the River Walk and Alamo is... a golf course?!? Even if it is accurate that the Palmer Course has the highest arithmetic mean for all of the submitted ratings in San Antonio, why would this matter to me? TripAdvisor knows I am not a golfer--I have made 101 contributions to TripAdvisor, and not one of them is for a golf course or shop.

One way sites like Yelp and TripAdvisor combat this problem is by showing me what my friends have rated. This is good, but I have friends who love golf. What I really want to know are the ratings of other people like me--it would be more informative to see the ratings of other museum-going, water-park-hating, jazz-loving non-golfing strangers who are similar to me. One of the benefits of furnishing ratings based on similar tastes is that it helps to filter out disingenuous raters. (Perhaps those who are paid or get discounts for the ratings they produce would like to see and rely on each other's ratings, but I rather doubt it.)

Another idea Jeremy and I discussed is discounting ratings from raters who only offer positive ratings. If a given user on Yelp or eBay only posts five-star ratings, wouldn't that call into question the authenticity of those ratings? Some may argue that people tend to write reviews when they feel strongly, but if this were true, we would see more ratings at both the top and bottom of the scale rather than the rating inflation exhibited on most sites. eBay data from 2007 demonstrated the median rating for "Item as Described" was 4.8 out of a possible 5; 67% of the ratings on Yelp are four or five stars; and virtually everyone who rates a video on YouTube gives it five stars.

This is when Jeremy offered the idea of using gamification to improve ratings. I am actually not as much of a fan of gamification as most social media pros. Even though I am an avid PC and mobile gamer, I just do not believe most people go through life wishing they could turn their communications, grocery shopping, dining, driving and TV viewing into a competition. In addition, too much gamification is designed to get people being inauthentic, distorting rather reflecting true opinion, influence and engagement. The fact a brand can amass a million followers by giving something away in a Zynga game is not sign of gamification's value to marketers but of its ability to falsify sentiment and destroy the legitimacy of Facebook "likes."

Jeremy's idea was one of the few I have heard that could turn gamification into a force for authenticity: What if reviewers were rewarded for offering R&Rs that most closely match a bell-shaped curve? Wouldn't reviewers' opinions count more to others if they reflected a more statistically accurate normal distribution of opinions? Perhaps discounts should be given not for any review (or positive reviews) but for people who create true value by creating the most believable and accurate content? If most reviewers were motivated to create R&Rs that were unbiased in a statistical sense, it would create greater dispersion of ratings, increasing the spread between the best and the worst products and services, and penalize (or remove the incentive) for paid reviews.

You probably have some of your own ideas for how R&R sites could be improved. The ease with which ideas can start flying as people discuss the problem is an indictment of how little has been done to improve the R&R process. Yelp says its ranking system "already factors in the number of reviews (and) whether they come from experienced Yelpers or first-time reviewers," but they've done little to make this evident.

If today's R&R leaders do not want to end up like Monster and MySpace, lapped by competitors with greater innovation and differentiation, it is time for the early market innovators to put as much effort into improving the usefulness and validity of R&Rs as they do into new marketing and advertising products. As Jeremy noted, "Trust is going to be the primary currency of the linked economy. If you lose it (and eventually you will get exposed), you're going to have a long road back."


Erica Driver said...

Here's how I tackle the issue of validity of online ratings and reviews:
1. I don't give too much weight to the overall rating a venue gets -- unless there is a huge number of reviews behind the number.
2. I don't even read the reviews that are five stars. I think of them as just cheerleading.
3. If I'm looking for peoples' experiences at a place (say, a restaurant or hotel) I will carefully read the 4-star-and-lower reviews. People who give a place 2-4 stars seem to have more thoughtful comments -- they tend to list both pros and cons.
4. I only read and pay attention to reviews posted during the last few months. Older reviews aren't as relevant.

Augie Ray said...


Thanks for sharing your approach. Good ideas, but it's sad that we live in a world where we discount five-star ratings because, sometimes, things deserve give stars!

I like your approach to older ratings. One of the things I'm doing recently is using the feature on Yelp that graphs ratings over time. I find it interesting to see what is trending up and what down.

Thanks for sharing!

Anonymous said...

How do you feel about anonymity on review sites?

Augie Ray said...

Great question, Jeffrey. I'm curious your thoughts on this.

On the one hand, there's the idea that people will be more honest if they can post ratings anonymously. But, as noted in my blog post, true anonymity creates the opening for inauthentic posts.

My feeling is that two-tier identity is the answer. Everyone who submits a rating should be forced to authenticate--this is the way the R&R sites can validate people are playing fair and not merely spamming fake positive ratings. But, I think the R&R sites should allow people to choose their level of anonymity.

I have no problem with R&Rs being posted by people who want to remain anonymous, provided the R&R site is doing due diligence to ensure the person is a real person posting real ratings.

What do you think?

Meritmeter said...

Great Article, actually we have been researching the issue for 5 years now and I read so many market research and Harvard business school thesis papers on the issue. Yes, it is a big issue but it be fixed as technology evolves, and that can be by using video reviews, with some smart algorithmic system in the background.

And we decided to build a new platform for genuine products reviews, we called it Meritmeter. (

In Meritmeter, we rely on posted video reviews where users see the reviewer and the product in their hands, they talk about the product and they rate it too. We calculate the user rating for the product (Low, Medium, High) and we aggregate all reviews ratings into one product rating, we take into account the number of reviews for that product (not arithmetic mean), the reviews ratings, the reviewer credibility score, besides we allow other users to vote on the review, and we take that into account too when we calculate the review score and the product score. We do not use 5-stars, as 1 star and 5 star are not trusted, we use Low, MED, High to make it easier to rate, read and compare products with colors.

We still a fresh startup but based on our research and findings, our solution will improve the quality of reviews authenticity and increase the end user satisfaction with their decision to buy a product.