Monday, August 11, 2008

Social Media ROI: Be Careful What You Promise (Part Two)

Yesterday on ExperienceTheBlog.com, I shared a concern that history is repeating itself. I am worried that Social Media practitioners are setting unreasonable expectations about the ease with which ROI can be ascertained, the same mistake made by Internet Marketers a decade ago.

In the infancy of the Web, Internet marketers sought to earn trust and a portion of the marketing budget by promoting the measurability of the Marketing Return on Investment (MROI) of online tactics. These promises worked; the Internet channel earned an ever larger share of the marketing pie, but the expectations that were set about measurability have become as much foe as friend. First of all, online marketers face a much greater expectation to produce immediate short-term results and make constant, incremental improvements than do those who work in traditional media. Secondly, with the exception of those who sell direct to consumers online, MROI has proven very difficult to establish.

I mention the history and challenges of online MROI because I fear Social Media consultants are in danger of repeating some of the mistakes of the past. Social Media Today and other sites are filled with posts focusing on the ROI of Social Media, but while I've read a great deal of theory and belief, I've seen precious little practical information.

In theory we can measure the increase in purchase intent or the loyalty of consumers engaged with social media, but in practice this is darn near impossible because Social Media is a complex, dynamic, and distributed strategy that is part of a complex, dynamic, and distributed marketing environment. With consumer perceptions and actions being shaped by hundreds or thousands of possible combinations of traditional, online, and social media, how can we ascribe a specific financial value to a single interrelated tactic out of many?

To illustrate my point, consider an automaker that launches a large marketing campaign that includes a mix of traditional, online, and social media:
  • National media is purchased across TV, print, radio, and online;
  • A microsite is launched;
  • The microsite is promoted via the brand's Facebook, Twitter, Identi.ca and other profiles;
  • Consumers ask questions or share criticism and praise about the automaker's products on microblogs;
  • The microsite invites consumers to join a community;
  • Community members are invited to a test drive of the new car when a branded event tour is nearby;
  • Community members are invited to play an online casual game;
  • A social media press release reaches automotive bloggers, and some comment on and link to the microsite;
  • A traditional press release results in national publications offering articles about the car model and the microsite;
  • Photos of the car model are shared on Flickr;
  • Viral videos of the car doing stunts are launched on YouTube;
  • Community members are furnished with a widget for their blogs and Facebook profile; and
  • Auto dealers take out local print ads, promote the car to their own email databases, display signs at dealerships, and train their sales representatives on the advantages of the car model, resulting in increased sales.
In the end, the automaker knows the number of cars sold, the profit margin, and that the marketing spend, so it is able to compute the Marketing ROI of the entire campaign. But how can the automaker differentiate the value of Social Media from other media in this campaign? How might the value of the YouTube videos be weighed against that of the branded game?

There are many metrics available, but monetary value to the brand cannot be derived from the number of visitors, repeat visits, comments, time spent with the media, referrals, or inbound links. Nor can MROI be extracted from the number of Twitter followers, number of views of videos and photos, number of community members, number of people who show up at the mobile event, number of times the game is played, or number of widgets served. These are fine gauges that permit you to compare and contrast different tactics, but these metrics do not add up to MROI because they don't reveal the contribution of Social Media to the sales that resulted.

It is tempting to suggest that we can measure the value of the online community by surveying the members to see if they purchased at a greater rate than did those who were not part of the online community. This approach is subject to several problems, but perhaps the greatest of these is that it ignores cause and effect: Were people more likely to purchase as a result of their participation in the community, or were they more likely to join the community because they were inclined to purchase the car?

If you are aware of other ways of measuring the MROI of Social Media, I would be very interested in learning them. After exploring quite a bit about the MROI of online and social media, it seems to me there are two good ways to calculate MROI, but both require financial resources and careful planning:
  • Pre- and post-action surveys: To eliminate the cause-and-effect bias, a brand might conduct pre- and post-action surveys. For example, in our hypothetical car community, we might measure consumers' propensity to buy at the time they join and then check again at a later date; an increase in their likelihood to purchase would tend to prove the value of the community.

    Knowing that the brand invested $150,000 in a community and that members who responded to the survey reported an increase in their average purchase intent from 70% to 80% tells us something but it doesn't tell us that the community produced positive or negative MROI. This can be accomplished only if we are able to place a value on purchase intent. For example, if there are 20,000 members and the brand knows that a ten-point increase in purchase intent is worth $15 per person, then this community produced $300,000 of value for the $150,000 investment.

    This approach is subject to several biases: People may have seen ads or read about the new car in automotive magazines in the period between the surveys, so this while this approach narrows the focus on a particular medium it cannot eliminate the impact of other media. Also, if you conduct the survey within the community, you are only reaching active members and not the share of folks--often a significant percentage of joiners--who register for the site but fail to return more than once or twice. And those who respond to the surveys may not accurately represent those who did not respond; in fact, there's reason to believe people who purchased or are more inclined to purchase are also more inclined to respond to a survey request from the brand.

    In addition to the potential biases, there are other difficulties with this approach. First of all, you need to know the specific value of your goals; ascertaining the dollar value of a point of brand awareness, loyalty, knowledge of a program or feature, or purchase intent is not easy. Secondly, not every social media tactic is as easy to measure as community membership. You know who is in your community, but how do you measure the impact of the branded game, the widget, or the Flickr gallery. Since people do not register for these interactions, it is difficult if not impossible to reach these consumers to collect their data.

  • Regression Analysis: Another way to measure MROI for Social Media and every other tactic that is part of a discrete campaign is to survey individuals to establish their change in behavior or beliefs and to collect data about the media/channels to which they were exposed. By using regression analysis, you can find those tactics that were most positively associated with the desired changes.

    For example, the brand may survey consumers and ask whether they purchased or whether the brand is one they'd consider; these are examples of the dependent variable. The survey may also gather info about the media in which they saw or interacted with the brand. Whether a consumer saw a print ad, a TV ad, a game, or a YouTube video or whether he or she interacted with the brand via a community or Twitter are all independent variables. A regression analysis will reveal the degree to which each of the independent variables were most associated with the dependent variable.

    While this approach provides a broad view of the relative value of different media, it has some substantial drawbacks. First of all, it relies on notoriously unreliable self-reported data; most people are not good documentarians of ads they saw. Secondly, this approach is not inexpensive. Lastly, data collection techniques can be biased by the way data is collected; if you only collect data online, the consumers who respond will be more likely to have seen online ads or joined online communities.
My objective certainly isn't to discourage the measurement of Social Media but to suggest we be very cautious about how we toss around the term "ROI". Unless we can define specifically (and not theoretically) how to determine $X of return for $Y investment in Social Media, the term "ROI" should not be used.

Here is what online marketers did wrong and the lesson Social Media experts should heed: Internet marketers said measuring ROI was easy when what they should have said was, "Online media provides some specific metrics that will permit us to measure and improve response rates, but because consumers interact with so many different marketing channels and tactics, partitioning the MROI for online tactics apart from all other tactics is no easier online than it is for traditional media."

So by all means, track the churn rate of community membership, the number of followers, the response rate to microblog promotions, and consumers' perceptions of their social media interactions; just don't call these ROI metrics. If we are careful not to over-promise the measurability of Social Media MROI and underestimate the time and expense to measure true MROI, we'll avoid repeating past mistakes.

No comments: