Share

How to Conduct True A/B Ad Copy Testing in Google

Published: March 13, 2018

Author: Caitlin Halpert

Share

This week, the 3Q blog takes on the topic of testing – one near and dear to all digital marketers. 
If you read my previous post on testing traps (or my full AdQ whitepaper on SEM ad copy optimization), you know there are a lot of gotchas you have to consider when setting up A/B ad copy testing. I’ll say it again – you should consider better ways of using your time, like testing betas or bidding or optimizing ad serving, but since proper ad copy testing can sometimes make an impact, I’ll teach you how to do it correctly. (If you’re going to go against my recommendations, you might as well do it right.)

First, let’s start with the basics: the test must actually be controlled (i.e. the ad text must be the only difference between the test and control), which means you must use Ad Variations. And the output of the test must be useful and inform future optimizations. If your end goal is purely to determine which ad is better, then it is not a good time to A/B test.

Step 1: Formulate a Question

This is probably the most straightforward step in the scientific method when applied to digital advertising. What are you trying to figure out through your test? Here are some examples:

• Which value proposition is most compelling to customers, A or B?
• Does including free shipping in the ad headline improve results?

Step 2: Create a Hypothesis

This step is critical – and often overlooked. What result do you expect based on what you know so far? Before you even begin your test, you should determine what metric you’ll use to evaluate the test, and which variation you expect to be the winner.
Metrics: Use your primary campaign KPI, typically an efficiency metric like CPA or a volume metric like profit.
Control vs. Experiment: In most cases, you are introducing a new experimental ad that you hypothesize will beat your existing control. While this is generally assumed, we recommend that this be clearly communicated within your team (and to the client).

Step 3: Design your controlled test

Ad Variations is the only method of controlled testing that 3Q recommends because it allows you to test language differences across multiple campaigns, it is simple to set up, and it allows you to still use the “optimize” ad rotation setting. Ad Variations also eliminates any impact from ad history without needing to relaunch fresh ads, which can increase CPCs. Any testing through a third-party bid management platform will have similar issues, so we’d still recommend using Ad Variations.

Step 3a: Determine your exact test language and location

Consider your hypothesis and find the best areas in the account to validate that point. Be sure any experimental ad copy also aligns closely with your hypothesis. There’s no need to apply a test broadly if you’ve designed a focused experimental question.

Step 3b: Determine your parameters

• Select your KPI: Determine which metric you will use to evaluate your results.
• Maximum and minimum test duration: Tests should always be run for at least one week and no more than four weeks to account for day-of-week differences and to ensure you’re not designing tests that will have limited value because of time restrictions.
• Confidence Interval: Set the level you’ll use to evaluate statistical significance of your results (90% is sufficient in most cases for digital advertising because fast tests are critical).

Step 3c: Prepare for a Null Result

Decide in advance what action you’ll take if you’re unable to find a winner during the planned duration. It’s important to consider this in advance so that you can move quickly at the test’s conclusion. This also helps you keep clear which decisions are based on the data, and which might simply be influenced by the data in the absence of significant results. Ultimately, you’ll still need to decide which action to take to drive the best performance – so plan for it! Under no circumstances should this null result plan be “let’s just run this test longer.”

Step 3d: Choose “Optimize” in the Ad Rotation Settings

Maintain the “Optimize” ad rotation setting across all campaigns, even when testing. In our agency-wide testing, we’ve found that the “optimize” rotation setting can lead to 10-20% more conversions than campaigns set to rotate evenly. Every single campaign should be set to “Optimize.”

Step 4: Analyze the results

This one’s worthy of its own post – which is coming up later in the series. We’ll talk about sample size and significance (practical and statistical) and how those work to determine whether you’ve actually discovered a winner.
 
Download our Introduction to SEM AdQ, 3Q’s approach to ad copy testing, for a start-to-finish look at how (and when) to put better ad testing into play.