What to do When Statistical Significance Isn’t an Option
Published: December 23, 2013
Author: Bob Sturges
We’ve all been there before: it’s week two of an ad copy test, and each ad either has just a few dozen clicks to its name, or CTRs and CVRs within a few thousandths of a percent of each other. What to do? You can’t let this run forever. Sometimes action needs to be taken without the luxury of waiting for statistical significance.
Below are some tips for what you can do when you reach that crossroads of data-driven rigor and “winging it”.
Go with what you prefer
Is it looking like a tie between two ads or landing pages? Or, is it clear that the test isn’t going to bear any meaningful results? In these situations, I will basically work with the client on determining what message or page is either more on-brand, or more logical.
For example, if a luxury brand is testing copy between an ad promoting quality merchandise and one promoting discounts, it would make sense to use the ad that is emphasizing quality goods. If there is a toss-up between sending users to the homepage or straight to the store page, but you feel one or the other better represents your company’s interests, then it’s time to just make a judgment call.
Hasten test results
Occasionally, you come across a test that feels like a winner, but is just getting no data. In these situations it can be worthwhile to briefly increase the traffic to the test. This could mean bringing the test page/ad up to a 50/50 split if it wasn’t already, or increasing bids to get more traffic. If you initially roll it out to only a few ad groups, expanding it to a larger portion of the account can bring in traffic more quickly. Sometimes it can be best to deploy the test conservatively if you don’t know results, but once you get a feel for expected performance bumping, up the traffic volume can be a relatively safe approach for reach statistical significance in a reasonable timeframe.
Design tests to be aggressive
One of the biggest ways to avoid the statistical significance quagmire is to design the proposed test in an aggressive fashion. If you know you have a low-volume account or campaign, an ad test that’s trying to gauge the impact of an exclamation point versus a period will be doomed from the start. Instead, you can focus on higher-impact testing such as price points, features/benefits, promotions, and other angles.
When it comes to landing page tests on a low-volume account, I would focus on pages that feature different templates and funnels rather than minor iterative copy changes across a single template. This may take more time to set up/create, but you won’t be wasting valuable time on a long, drawn-out test.
At times, tests with no clear outcome are going to be unavoidable. In these cases, it behooves the account manager to know when to cut their losses and move on. The opportunity cost of waiting out a low-impact test can often significantly outweigh the gains you may receive from eventually determining a winner.