Tuesday, April 1, 2014

The Problem With Social Media Case Studies

We all love a good case study, don't we? Especially in the social media business, where results can be difficult to measure and prove, there is nothing more validating than knowing someone else has succeeded and can show us the way.

Problem is, despite years of experience and thousands of case studies promoted by agencies, consultants and speakers, most companies are still struggling with their social media strategies. How is this possible when there are so many good case studies to be found?

One problem with case studies is the way marketers use them. There has been a tendency to latch onto any successful case study as an archetype for success, but case studies are not best practices. For example, I can present a case study on how dropping out of high school led to success for people like Walt Disney, Sir Richard Branson and David Karp, but that does not make skipping school a good idea. The same is true for many social case studies; social media professionals may love to learn how Coke or Starbucks succeed in social media, but how many brands can leverage the existing marketing spend, reputation and consumer affinity of brands such as those?

Several years ago, a consultant was pitching my team at a financial services company and offered Dell Outlet's Twitter strategy as a relevant case study. I stopped her and asked how Dell's use of Twitter to sell refurbished hardware was pertinent to us in the banking and insurance business. The consultant was unable to answer that, and it made me wonder how many times that case study had caught the attention of marketing professionals who had absolutely no opportunity to use that knowledge within their own business or vertical. Who cares how much money Dell Outlet made on Twitter if your company cannot promote reconditioned product via a Twitter account?

Aside from how they are used, many case studies suffer from one or more problems that limit their value. First, most are produced to promote an organization's or person's products or services. This is not necessarily a bad thing, but it should cause us to approach the information with a healthy dose of caution and to question if we are getting the full and complete story.

Second, most case studies, particularly in social media, do not close the loop in terms of both costs and benefits. It is rare to see case studies with dollar signs--either for investments or returns--and this means most case studies are less business examples than they are entertaining stories. After six years dedicated to social business, I cannot stand to see one more case study measured in new fans or retweets. (And neither should you!)

My greatest concern with case studies, however, is that most simply do not hold up to any sort of scrutiny. Hungry for evidence of social media success, marketers have been far too eager to drink the social media Kool-Aid.

Take, for example, Dell, which everyone widely recognizes as an organization that has transformed itself into a model social business. The brand famously initiated social listening and customer care in response to the firestorm created by Jeff Jarvis's June 2005 blog post "Dell Lies. Dell Sucks." Since then, the organization has been a model of social innovation--Dell made millions with its @DellOutlet Twitter handle; it launched Dell Ideastorm to co-innovate with consumers; it deployed a social listening command center; and it used social listening to recognize and immediately resolve concerns with XPS 13 pricingIn 2011 when Forrester went looking for the most socially mature "Empowered" companies, Dell was an obvious choice for the short list.

Just one question: Where is the evidence of Dell's success? Since June 2005 when Jarvis posted his criticism of Dell until October 2013 when Dell went private, the company's stock dropped 66%. Over the same period, other tech stocks performed much better: Hewlett Packard stock was even, Cisco was up 17%, IBM up 138% and Apple (one of the least social businesses around) was up 1249%. What, exactly, makes Dell such a worthwhile and repeatable example of social success (other than really, really good PR)?

Many other well-known case studies collapse pretty easily when you dig a little deeper. I was quite critical of Esurance's Super Bowl sweepstakes a few months ago because so many bloggers and journalists went gaga for the big numbers reported by the brand. (Why is it when accountants see suspiciously high numbers, they question them, but when marketers do, they immediately tweet, share and blog about them?) I questioned the business and brand value of the tweets made to enter a social sweepstakes, but some people criticized my analysis by pointing out Esurance boosted its follower count from 10,000 to more than 260,000.

It's worthwhile to look at how the brand is doing almost two months later: The account has since lost half its new followers. Even more damning is this: Esurance started promoting its upcoming sweepstakes via its Twitter account on January 27. In the prior seven days, with just 10,000 followers, the company tweeted 12 times and received an average of 14 replies, 9 retweets and 71 favorites. In the last seven days, despite having 1300% more followers than before their sweepstakes, the account has tweeted 25 times and received an average of 4 replies, 3 retweets and 3 favorites. Following the brand's $5 million program, the Esurance account is getting less engagement. Why? Because those new followers are not prospects, advocates or customers; they were nothing but random people who wanted to win an easy $1.5 million!

It isn't just positive case studies that suffer from problems; some of the most well-known negative case studies do not stand up to evaluation, either. As I have written in the past, some of the most famous "social media crises" left no signs of adverse business impact. United Breaks Guitars has been cited thousands of times as proof of how a social media PR crisis can damage a brand, but if it did, there is little evidence to be found; in the six months following the release of the "United Breaks Guitars" video, the company's stock outperformed competitors Delta and US Airways by more than 150%.

How about the backlash that Chick-fil-A faced after people objected to comments from its CEO about the biblical definition of the family unit?  Consumer use of the chain was up 2.2 percent in the quarter following his comments, market share rose 0.6 percent, and total ad awareness was up 6.5 percent.

Or what about the Bank Transfer Day backlash against Bank of America's debit card fee? While credit unions reported picking up $4.5 billion in new deposits, Bank of America saw average deposit balances rise 500% more--an increase of nearly $25 billion in the same fiscal quarter as Bank Transfer Day. As for BoA's stock, in the six months following Bank of America's disastrous fee announcement, shares rose 51% while the Dow Jones increased just 22%.

United Breaks Guitars, Chick-fil-A and Bank Transfer Day are oft-cited examples of social media crises that damaged companies, but as case studies, they are pretty toothless. When in the midst of a social PR problem, it is easy for bloggers and journalists to attract clicks, make unsubstantiated conclusions and declare the situations to be cautionary case studies for others, but how many of these same authors go back months later and look for the short- or long-term impact?

Good case studies are woven from reality and facts and do not disintegrate the moment you pick at one thread. Demand and question more, and we can begin to separate the wheat from the chaff.


Robert said...

I like this. Thank you for sharing it. In the classroom, for example, we seek case studies that define process and reveal persuasion theories that fit the situation of each case study. We don't care much if they are good or bad practice. This illustrates one of my peeves - PR / marketing case studies and pseudo-research (surveys, etc.) that pose as sincere efforts, but are merely disguised marketing efforts. Thanks for this. We'll discuss in class.

Augie Ray said...

Thanks for the comment, Robert. I love knowing this blog post may inspire some classroom discussion. As a former Forrester analyst, I share your concern about "studies" and "case studies" produced by companies with a horse in the race and a goal of selling more products and services. Of course, that does not necessarily mean the data or lessons are wrong, but it certainly means no one should accept it all at face value.

Thanks for the note!

Alex Roe said...

I like this too - good common sense. Case studies are interesting but the benefits of social media in terms of ROI are not easy to come by. And as you note, social media disasters aren't always as damaging as everyone claims they are.

Maybe the costs and benefits of social media use need to be looked at more closely.

Augie Ray said...

Alex, thanks for the comment.

"Maybe the costs and benefits of social media use need to be looked at more closely." I agree completely, except for the "maybe" part! :)

Tom Snyder said...

I'll probably get excommunicated from the Social Media club for saying this, but maybe Social Media just is not a very good marketing tool from an ROI standpoint, and because there are so few real success stories, it needs to be oversold by its purveyors by finding (and even still exaggerating) the rare exceptions to the growing body of evidence.

For the record, I said "Maybe." ;)

Augie Ray said...

Tom, thanks for the dialog. I am coming to the conclusion that social media is NOT a marketing channel. It probably never was, but at least an argument could be made at a time when brands could count on a certain level of organic earned media. But those days are gone, and brands keep trying to be talkable with dumb viral videos and inane sweepstakes.

There are exceptions, of course, but by and large brands have missed the meaning of the social era. It wasn't that marketers now have yet another channel to exploit but that the things already important--like customer experience, purpose/mission, loyalty, customer centricity and the like--are now even more important.

I think it's time for the stream of uninteresting spam and useless marketing content to end. It's time brands were more responsive in social, built social into their product and service strategies and focused more on loyalty than acquisition via social.

Augie Ray said...
This comment has been removed by the author.
Dennis jenders said...

Robert - I teach a class as well and would love to hear more about the case studies you are sharing.

Wendy Soucie said...

Well, I am glad that you went back and checked the numbers on those case studies. In the B2B industrial space, we can listen and learn from the hype, crash, burn, and lack of success and steer our big "iron ore boats" on the right course in social. It is worth our best efforts and late entry in the social business space to deliver what our customer need for their business success.

Augie Ray said...

Wendy, I often say that B2B is a place where brands can succeed with content strategies. In the B2C space, success has to measured in tens and hundreds of thousands, but for B2B, a handful of qualified leads can be a huge win. Some of the best (and more realistic) case studies I have seen have been in the B2B space!