The importance of creative de-bias in TV measurement

In the remnant TV market, advertisers typically place media buys on a weekly basis and have no guarantee of clearance. Furthermore, spots aired are seen by all viewers, as opposed to online ads in digital marketing in which random samples of viewers see different ads. This makes it difficult to execute a perfect creative test construct. What usually happens is that different creatives end up being distributed across different networks, and occasionally with different spend. Furthermore, with the introduction of new creatives in the mix, advertisers often encounter a scenario in which some creatives aired on certain networks, while the newer ones did not (or vice versa). The impact of network-rotation variability and imperfect creative split should therefore not to be ignored, as it can greatly affect the measured performance of creatives. This is what we refer to as ‘creative bias.’

In the illustrative graph above, Creative A outperformed Creative B, but Network A also outperformed Network B. When creative bias occurs, as it would in this case, it is hard to tell whether a certain creative outperformed the other because of the difference in their messaging, or because they aired on different networks and rotations. To allow for an apples-to-apples comparison between TV creatives, it is therefore important to de-bias the effect of such factors.

At Tatari, we have implemented a model that de-biases performance in two major steps. First, comparisons between each pair of creatives are made across every network on which these creatives aired. This step is essential as it informs our model which attributes to weigh more heavily in the calculation of final performance. Second, differences in spend distribution are then normalized to isolate the effective impact of creative performance. This allows us to compare creatives that aired on different networks with different amounts of spend.

Once our model completes these steps, we report creative performance in form of adjusted efficiency.  Adjusted efficiency shows advertisers how their creatives performed while minimizing the biases due to spend distribution inequalities. As mentioned earlier, it is worth noting that this bias is an inherent complexity of TV advertising and the related nature of media-buying. A typical A/B testing is therefore not possible with linear TV—those advertisers who present creative performance results without de-biasing can sometimes end up with incorrect results.

Mike Swinson

Chief Data Scientist.  I'm an explorer: I seek out the facts by following wherever the data leads.

Related

Streaming TV is the Future. But the Future Isn’t Always Tomorrow.

Streaming offers unique advantages, but the medium hasn’t matured enough to beat digital on precision, or traditional TV on scale.

TV Device Graphs Are Not (Yet) Ready For Prime Time

In a mere five years, TV advertising has turned from an offline into an online medium. This shift has brought challenges and opportunities to TV advertising measurement.

The importance of incrementality in TV

While search and social media – Facebook, in particular – are still the most popular, TV advertisers can be found on just about any platform, including radio, direct mail and out of home.