Strategic experimentation can leverage measurement techniques of market testing, modeling and even pre-post analysis. Each has its own advantages as outlined below.
The ideal methodology is market testing where you can vary specific aspects of your marketing for comparison against your “business as usual” marketing plan. Strategic experiments can be market tested in the controlled environment of direct marketing, where you can use random A/B list splits to compare the impact of offers or product positioning to a limited number of contacts before launching a full-scale campaign in mass media. Market testing also works very effectively for assessing mass media or integrated multi-channel marketing using specific sets of geographic markets that have similar characteristics and performance trends.
Wondering if your marketing budget is too low? Set up your test with incremental spending in just a small set of markets while decreasing your spend in other markets. Wondering if your outdoor billboard advertising is worthwhile? Try three different levels of outdoor ad spend in three sets of comparable markets while all other marketing activity is held constant. In each case, you are executing your marketing using different plans that are all viable, and learning at the same time.
A basic market test example might look like this:
Control – Business As Usual
Treatment – Incremental Brand Media
|Standard Marketing Plan||Standard Marketing Plan|
+ 20% budget in brand support for 6 weeks
|Markets – Phoenix, Chicago, Jacksonville*||Markets – Dallas, St. Louis, Atlanta|
* Note: “Business as Usual” marketing goes to all of the US but only these markets shown are used as the control group based on having sales patterns very similar to the test markets.
An incremental ROI analysis is conducted using the results from the test to compare the profits from incremental sales (treatment over control, adjusted for market size differences) to the incremental budget invested. If the results are positive and exceed the ROI threshold, the test shows the need for incremental budget.
More sophisticated testing based on modeling and design of experiments can provide broader learning as you test many variables of your marketing strategy at once. This approach works better for determining the mix of media channels and contact intensity since media can easily be varied in many different markets at once. It is not really viable for testing many different product positionings or offers concurrently in different markets since the spillover can be confusing and disruptive to customer perceptions.
There are often many questions as to the right marketing mix in terms of the media channels, the level of spend, touchpoint timing, and the budget spread across branding, demand generation, direct marketing, channel partners, and direct sales. A well-structured test design with numerous variations in the marketing mix and intensity by market provides rich data for modeling the variables to outcomes such as sales volume, customer acquisition, customer retention, or customer value. The entire marketing spend is a combination of a full-scale campaign and extensive test. The modeling can guide shifts in budget allocation and marketing integration. The richness of the measured results can move the organization much closer to optimizing its marketing ROI.
Some marketers may not have the opportunity for market testing or modeling based on small market coverage, data limitations, or complex marketing environments so they may be limited to a pre-post measurement. Pre-post measurements use the prior period without marketing to calculate a baseline for average sales volume against which the period during or following the marketing campaign is compared to determine the lift. While this method provides directional results, it does not have the reliability of market testing or modeling.
Strategic experimentation using pre-post measures requires that the variables and alternatives are introduced at different time periods to compare the sales lift of each different strategic initiative. If this method were used for examples above, the marketing budget would be increased or the marketing mix varied for a short period of time and tracked. You are less likely to test the more extreme changes to your marketing since you are experimenting with all or most of your marketing contacts, seeking a significant lift in performance. Because the methodology does not eliminate the influence from external factors, such as competitive marketing, seasonality, or economic fluctuations, even positive results should be re-tested and adopted with caution.