Saturday, March 08, 2014

Tyranny of Experts - Consumer Reports Edition

How often have I needed to replace an appliance and just gone to Consumer Reports, picked one of the top couple of models (usually the top one), and bought it sight unseen, happy in the knowledge I had bought the best - and with little effort?  Today, when I went online to decide on a vacuum cleaner,  I realized what a mistake that can be.

Consumer Reports gave the highest rating to a Kenmore, saying it is "impressive" and the "top-pick:"

I found it online and was about to purchase, but then looked down at the user ratings and saw that users gave it only 2 stars out of a possible 5.  Worse, 75% of actual users would not recommend it to a friend.  Here is the summary:

And here is a typical user review:

It took me about 5 minutes of scanning the user reviews to decide not to purchase the Kenmore.  And to be honest, I was pretty shocked.  Despite what the experts said, regular people seemed to hate this unit.

So I started skimming down the page to look for models that users themselves seemed to like.  I found a Miele, which was given mediocre ratings by the experts at Consumer Reports.  But actual users gave it 4.4 out of 5 stars:

And here is a typical user review of the Miele:

Now I'm not saying the Consumer Reports experts aren't smart.  I am sure they are.  And in some cases, their ratings agree with those of users.  The problem is that their own scorecards for quality are based on factors that are not well aligned with what consumers themselves actually care about.

Will I continue to subscribe to Consumer Reports?  Probably.  Will I listen blindly to their recommendations?  No.  From now on I will start with feedback from consumers - and then, if I have time, I will read the expert reviews.

[This post elaborates on a piece I wrote earlier for the Center for Global Development.  For more on the topic of feedback loops, citizen sovereignty, and development, see my upcoming review of Bill Easterly's new book, The Tyranny of Experts. Or better yet, buy the book itself. ]

Share on Facebook