The importance of creative de-bias in TV measurement

In the remnant TV market, advertisers typically place media buys on a weekly basis and have no guarantee of clearance. Furthermore, spots aired are seen by all viewers, as opposed to online ads in digital marketing in which random samples of viewers see different ads. This makes it difficult to execute a perfect creative test construct. What usually happens is that different creatives end up being distributed across different networks, and occasionally with different spend. Furthermore, with the introduction of new creatives in the mix, advertisers often encounter a scenario in which some creatives aired on certain networks, while the newer ones did not (or vice versa). The impact of network-rotation variability and imperfect creative split should therefore not to be ignored, as it can greatly affect the measured performance of creatives. This is what we refer to as ‘creative bias.’

In the illustrative graph above, Creative A outperformed Creative B, but Network A also outperformed Network B. When creative bias occurs, as it would in this case, it is hard to tell whether a certain creative outperformed the other because of the difference in their messaging, or because they aired on different networks and rotations. To allow for an apples-to-apples comparison between TV creatives, it is therefore important to de-bias the effect of such factors.

At Tatari, we have implemented a model that de-biases performance in two major steps. First, comparisons between each pair of creatives are made across every network on which these creatives aired. This step is essential as it informs our model which attributes to weigh more heavily in the calculation of final performance. Second, differences in spend distribution are then normalized to isolate the effective impact of creative performance. This allows us to compare creatives that aired on different networks with different amounts of spend.

Once our model completes these steps, we report creative performance in form of adjusted efficiency.  Adjusted efficiency shows advertisers how their creatives performed while minimizing the biases due to spend distribution inequalities. As mentioned earlier, it is worth noting that this bias is an inherent complexity of TV advertising and the related nature of media-buying. A typical A/B testing is therefore not possible with linear TV—those advertisers who present creative performance results without de-biasing can sometimes end up with incorrect results.

Mike Swinson

Chief Data Scientist.  I'm an explorer: I seek out the facts by following wherever the data leads.

Related

Frequency capping in streaming TV campaigns

Being able to target customers with the optimal number of views and minimize CPV (and CAC) is a unique advantage of streaming TV, yet the opportunity gets squandered.

Why your TV commercial should have a persistent brand watermark

What exactly makes a great ad? One creative element has been effective for 70% of Tatari clients who have tested it: brand watermarking!

Measuring streaming TVs: it takes two to tango

Measuring streaming outcomes is only half; measuring the quality of the impressions delivery is the other half.  This is the first in a series of articles on the topic.