Sunday, July 20, 2008

Measuring Engagement and Marketing ROI, Part 2

Yesterday on this blog, we began an exploration of how to measure engagement and ascertain marketing ROI. The limitations of Exposure and Action Tallies were reviewed, and we ended with a brief summary of Transaction Metrics, which work well for direct response programs but aren't an option for marketing efforts that lack an immediate purchase or other call to action. For these sorts of marketing programs, different engagement measurements must be explored.

Content Engagement Measures: For programs without a transactional purpose, one way to measure engagement is to track how often key content is viewed. This is accomplished easily online since typical web analytics reveal how often a particular page or section of the site is viewed, how many pages were visited, and common paths through the site. This sort of measurement can be used to gauge which content consumers find most appealing and which content draws consumers into deeper brand engagements.

This type of measurement is most typically used online but can also be implemented within event footprints. With the use of passive crowd measurement techniques such as video tracking, RFID, and other means for monitoring physical movement, event marketers can evaluate if consumers are pausing within key content zones as they pass through an event footprint.

While Content Engagement measures are valuable for evaluating different content, design, and promotion alternatives within a particular medium, they provide no means for comparing engagement across media.

Blended Metrics: Some smart marketing experts have offered up new blended metrics that attempt to derive an objective engagement value by combining different sorts of web measures into a single calculation.

This one comes from Eric T. Peterson, and it combines Content (click depth, duration), Action (recency of repeat visits), and Transaction (Feedback) into a single calculation. Visit his blog for a complete explanation:

A simple blended metric was suggested by my friend, Jeff Larche, on his blog, Jeff's Content Interest Index is calculated by adding instances of a page’s “Printer Friendly Format” and “Email a Colleague” actions divided by page views.

While these are fine attempts to derive some sort of objectivity out of diverse Web statistics, these sorts of approaches fall short of measuring true engagement. First of all, they are manufactured values prone to the same biases and problems as the component statistics from which they are derived. Also, these are Web-only metrics that do not provide any basis for comparing engagement across media. As with the other measures previously mentioned, these are best suited for split tests or tracking improvement over time and not for comparing dissimilar marketing programs.

Brand Lift Metrics: For marketing programs with non-transactional goals, the best but most difficult statistics are those that measure brand lift. Depending upon the goal of a program, this could include aided and unaided brand awareness, brand preference, purchase intent, or desired brand associations.

Measuring brand lift is more complex and expensive than other forms of engagement measurements, requiring pre- and post-program surveys or the use of control groups to determine the effect a particular program. One of the greatest challenges can be to track or find consumers who've been exposed to a particular program or marketing communication; how can the people who see a print ad or visit a site without registering be located and surveyed?

ROI: The Only Engagement Metric That Matters: With the exception of direct response programs specifically geared toward purchases or other transactions, every category of marketing measure fails to assess engagement. There is no single magic yardstick that will permit marketers to compare the value of a print ad versus a microsite versus a live event.

Of course, we can't merely throw up our hands and tell our CFOs to get off our backs. Accountability and ROI are the hottest topics at marketing conferences these days, and the focus on marketing results isn't going away. Despite all the buzz about marketing ROI, the situation is pretty darn poor; Ad Age reports that nine of 10 finance executives say they don't use return-on-investment metrics to set marketing budgets and just one in 10 marketer respondents say they could forecast the effect of a 10% cut in spending.

I have no secret recipe, but I'd suggest the way we marketers can improve our ability to measure ROI is to do the following:

  • Specifically define the goal of a particular marketing investment.
  • Ignore every metric that isn't laser focused on that goal.
  • Focus on only those metrics that matter.
  • Involve your financial peers in the creation of an ROI model.
  • Plan and budget to execute not merely the marketing program but also the measurement program.

Sound easy? It isn't--not by a long shot. Say your goal is to increase unaided awareness of your brand; what is the dollar value of a one-point increase in unaided awareness? You'll need to know this in order to ascertain if your $500,000 investment that increased awareness five points represents positive or negative ROI.

The ability to assign value to marketing outcomes is vital for ROI calculations. If your goal is to increase traffic to your Web site, you must assign a dollar value to each site visitor (or perhaps different dollar values to different site visitors). If your goal is to obtain additional consumer records for your permission marketing database, you'll need to determine a dollar value for each individual added. No matter if the objective is to increase market share, increase consumer awareness of a particular brand attribute, or spark Word of Mouth in social media, we must first understand the value of the outcome in order to establish an ROI model.

The Limits and Distractions of ROI: I'll admit that part of my concern with today's obsession with marketing ROI is that it cannot be determined with the same ease or in the same manner as the ROI of an investment. Investments are discrete--you can easily track the impact within your portfolio of each specific security--but marketing programs aren't discrete; they're chaotic.

If a consumer sees a print ad, then later searches and clicks a link optimized via an SEO program, surfs through a Web site, opts in for future email communications, and months later responds to a marketing email, how do we attribute value to each of these components, none of which engages the consumer in a vacuum? Is it reasonable in today's highly-niched, media-fractured, multi-channel, rapidly-changing, consumer-controlled marketing environment to expect marketers to have more mathematical acumen than chaos theorists?

Furthermore--if you'll permit me to turn philosophical for a moment--how does one measure love? As marketers strive to create strong emotional bonds between consumers and brands (or lovemarks, as Kevin Roberts call them), how is that to be captured in cold, hard numbers. What is your love for your spouse worth compared to your children, parents, or best friends? And what is the value of your love for Disney, Harley-Davidson, Coca-Cola, or the brands you hold near and dear to your heart? Must everything be measured in cents rather than sense? Would you trust a marriage counselor who tried to objectively measure progress by counting hugs and kisses rather than the feelings and perceptions of the troubled couple?

These are exciting times for marketers with a dynamic ever-changing environment and new methods for measuring and validating the work we do, but we cannot let the drive for measurability get in the way of good marketing. The lack of an easy measurement methodology shouldn't prevent us from exploiting great marketing ideas, because if we avoid programs that cannot be weighed, probed, surveyed, studied, and measured in detail, we will miss opportunities.

If the only tool you have is a hammer, you tend to see every problem as a nail. There are too many interesting and valuable screws and bolts in the marketing world to be ignored!


Jeff Larche said...

Augie, thanks for giving my Content Interest Index a plug! I encourage everyone who is responsible for managing content (or especially those managing those who manage content!) for a large web site to install this simple web metric. It's a terrific coaching tool.

I've enjoyed both parts of your long posts, Augie. Well-researched and organized!


KDPaine said...

Great post! really good points. We're ( doing some very interesting work with non-profits that use Google Analytics to correlate proactive social media efforts with online donations But I would also argue for another way of looking at the "R" and "I." First of all, alot of the time the "I" is pretty small -- and that leads to skepticism about the numbers. If you spend $500 on a podcast and get sales in the tens of thousands it's great news, but doesn't compare well with anything traditional. The other problem that you touch on but no one has solved is the isolation factor -- how do you know whether its the blog posting, the tweet, the podcast or the banner ad that results in the sale.. At the moment, most people attribute it all to the banner ad because that's what's being measured.
Finally, what about money saved? If Best Buy is reducing its turnover by xx% -- the "R" is actually lower recruitment and training costs. Too often marketers forget that expenditures avoided are just as important a return as revenue.

Paul Hender said...

Hi Augie,

Great post and I agree with your point about the sometimes misplaced obsession with quantifying ROI. One of my issues with ROI is the way that the metric is usually expressed as a ratio. As Katie points out, a very small investment that produces significant engagement can yield such large ROI figures that they can almost appear to lack credibility – ROI of 10,000% anyone? To take it to its natural conclusion, the best way to maximise ROI is to invest nothing, since anything divided by zero is infinity! Clearly this is nonsense.

However demonstrating the contribution that a particular marketing program makes to the success of an organisation makes sense. While isolating the effect of say a PR programme against the background noise of other marketing activities (and other effects) can be difficult, it doesn’t mean that it can’t be done. At Metrica ( we have used established econometric techniques to demonstrate the contribution of both mainstream and social media to among other things website traffic, product sales and charity donations. Similar techniques can also be used for less transactional measures such as awareness, message pickup, dare I say it even love (given brand advocacy can be picked up with market research approaches).

One of the complaints against social media measurement is that because there are so many potential metrics, it is difficult to compare the results with more traditional forms of marketing. However the benefit of econometrics is that the inputs are essentially dimensionless – anything that can be quantified can be put in: money spent, eyeballs, number of posts, type of conversation, ranking, Eric Peterson’s CRDLBFIS score…even the weather!

Augie Ray said...

Thanks to Jeff, KD, and Paul for the feedback and interesting ideas.

KD, my agency, Fullhouse, has a long-term relationship with a non-profit for which we do both pro bono and discounted work. If you'd care to share any info on your work with non-profits, I'd be happy to share this with the people responsible for these efforts. You can reach me at "aray at".

And Paul, I'd love to understand more about Metrica's econometrics approach, especially how you correlated weather into the equation. :) Seriously, I am always searching for interesting topics and good topics for this blog, and if you care to share more infom please do so at the email address noted above.

Thanks for the feedback, everyone!