The importance of creative de-bias in TV measurement

May 30, 2018

In the remnant TV market, advertisers typically place media buys on a weekly basis and have no guarantee of clearance. Furthermore, spots aired are seen by all viewers, as opposed to online ads in digital marketing in which random samples of viewers see different ads. This makes it difficult to execute a perfect creative test construct. What usually happens is that different creatives end up being distributed across different networks, and occasionally with different spend. Furthermore, with the introduction of new creatives in the mix, advertisers often encounter a scenario in which some creatives aired on certain networks, while the newer ones did not (or vice versa). The impact of network-rotation variability and imperfect creative split should therefore not to be ignored, as it can greatly affect the measured performance of creatives. This is what we refer to as ‘creative bias.’

In the illustrative graph above, Creative A outperformed Creative B, but Network A also outperformed Network B. When creative bias occurs, as it would in this case, it is hard to tell whether a certain creative outperformed the other because of the difference in their messaging, or because they aired on different networks and rotations. To allow for an apples-to-apples comparison between TV creatives, it is therefore important to de-bias the effect of such factors.

At Tatari, we have implemented a model that de-biases performance in two major steps. First, comparisons between each pair of creatives are made across every network on which these creatives aired. This step is essential as it informs our model which attributes to weigh more heavily in the calculation of final performance. Second, differences in spend distribution are then normalized to isolate the effective impact of creative performance. This allows us to compare creatives that aired on different networks with different amounts of spend.

Once our model completes these steps, we report creative performance in form of adjusted efficiency.  Adjusted efficiency shows advertisers how their creatives performed while minimizing the biases due to spend distribution inequalities. As mentioned earlier, it is worth noting that this bias is an inherent complexity of TV advertising and the related nature of media-buying. A typical A/B testing is therefore not possible with linear TV—those advertisers who present creative performance results without de-biasing can sometimes end up with incorrect results.

Mike Swinson

Chief Data Scientist.  I'm an explorer: I seek out the facts by following wherever the data leads.

Related

Tatari Selected As Only TV App Partner In Shopify’s Plus Certified App Program

Tatari is proud to be the only TV app within the new ad tech category of the Shopify Plus Certified App Program, which was launched to promote best-in-class partners for merchants on the Shopify platform. To date, Tatari’s best-in-class Shopify app has helped more than 200 Shopify customers easily access TV advertising with a simple integration.

TV Ads Drive Mobile-App Re-engagement

Read our white paper to see how Gametime, the leading mobile ticketing marketplace for last-minute sports & entertainment tickets, used TV ads to drive existing app users to re-engage with their app.

There Will Be No Kingmaking in the New World of TV Measurement

After decades of a “business as usual” approach to TV measurement, the industry has at last broken free of outdated paradigms and Nielsen’s quasi-monopoly on TV measurement.