By using this Site, you consent to the use of cookies. For more information, please see our Privacy Policy.


Experiment to Refine Your Attribution Solution

Volume 3, Issue 1 - January, 2013

Parameshvyas Laxminarayan, Director, Product Management, Visual IQ

Humans make numerous decisions in their day-to-day life with a clear intention or goal in mind - the outcome of which has great bearing on the next action they’ll take. Experimental studies in cognitive science reveal how the decision-making process is directly associated with the probability of obtaining reward. This behavior-reward probability is learned by solving what is called the credit-assignment problem. Though comparisons to cognitive neuroscience aren’t commonly made, digital attribution solves more or less the same credit-assignment problem by studying consumer engagement with media assets and the degree of influence those assets have on the purchase decision.

Attribution Experiments

By calculating and assigning credit to each marketing touchpoint along the purchase path, attribution helps marketers understand the influence various media channels, publishers, campaigns, creative elements, keywords and other such factors have on a purchase or conversion. This insight can then be used to build a more effective advertising strategy.

While the data set cast by attribution is usually very wide, let’s look at how to design a simple attribution experiment that analyzes user interaction with online display ads and how they influence consumer behavior. Specifically, the experiment will seek to understand which creative message produces the most conversions.

Designing Controlled Experiments

The fundamental idea of a controlled experiment is to compare data groups that are virtually identical. In the controlled mode, one variable is changed (test group) while the other (control group) is left undisturbed. The results obtained from the test group are then compared to the control group to measure the impact of that change.

To isolate the influence of an ad’s creative message on conversions, a controlled attribution experiment would be set up as follows:

  1. Define the test group: Start by collecting data for all of the variables that may have an impact on conversions and non-conversions. When evaluating display ads, the most obvious factors are size, placement, creative message and impression frequency, but there are many other variables to also consider, such as what website most consumers come from before seeing the ad, where they most frequently go after, and how long it took (in days and in interactions) on average to convert. Use these data points to define the test group.

  2. Build the control group: The control group should be virtually identical to the test group, except for the one aspect whose effect is being tested, in this case, the ad’s creative message. For instance, if different creative versions of the display ad appear on in the test group, then the control group should also include, with the original creative, in the same size, delivered at the same frequency, with the same average time to conversion, and so on. With two virtually identical environments, you can now associate changes in user behavior directly to different creative messages with greater accuracy and validity.

  3. Create the test environment: There are two testing approaches: A/B testing and placebo testing. A/B testing analyzes two or more realistic values to compare performance. In a placebo environment, the variable of interest is replaced with a nonsensical value know as a “placebo.” For example, when using A/B testing, the call to action on the display ad might change from “Offer ends today! Use code A1” to “Limited time offer! Use code B1.” Using placebo testing, the call to action might be replaced with an image or eliminated altogether.

  4. Test and measure: Next, the test version of the creative message is pushed live and its performance is compared to that of the control group. If replacing the creative in the test group results in a significant drop (or increase) in conversions, but the performance of the original creative in the control group remains largely unchanged, then you have a good understanding of its impact on user behavior. These insights can then be used to build a more effective advertising strategy.

While this example demonstrates how to conduct a simple attribution experiment, most marketers aren’t looking at a single data point like creative message, but rather at data from multiple sources. The sheer volume of data makes it virtually impossible to get a holistic view of marketing performance without time consuming manual data collection and manipulation. That’s where cross channel attribution solutions come in. By integrating dozens of channels and multiple terabytes of data, attribution solutions enable you to compare the performance of every channel, campaign and tactic, across your entire marketing mix.

So take a leaf out of the cognitive neuroscience field. Use attribution to better understand the decision-making rationale within the consumer’s mind, and use those insights to significantly improve your marketing effectiveness.

Subscribe to the IQ Advisor

Click here to subscribe to The IQ Advisor monthly e-newsletter.