The Benefits of Testing

I try to run a test in EVERY campaign I run. This way even if we lose, we win. We learn which variables have the greatest impact and apply results to future campaigns. Whether it is a list, an offer, or some part of the creative - I test. You should too.

One thing I've learned over the years is to never test ‘whispers’. Clients often want to test something that really won't make a difference or something that you can't rollout. Don't do it. Test only things that have a big impact. Test lists first, then offers, then creative – as this is the order of impact.

There are two types of tests, A/B and Multivariate. A/B is testing only one thing at a time. Multivariate is testing multiple things at once. Multivariate is more complex, but generates learning faster. Basically you make a grid (like an excel chart) and test one thing along the x axis and another on the y axis.
As you can see in this example, I'm testing two different lists against two different offers. I've split my 40k file into equal parts of 10k each. We then sent the direct mail piece to each cell. When the replies came in, we found that List 2 and Offer 2 both outperformed List 1 and Offer 1. In fact, the best result comes when we combine the two efforts. We will rollout our direct mail to the second list with the second offer.

Now of course, what we should really be looking at is the return on investment (ROI) from the cells (as the second list and offer may be more expensive than the first list and offer) but that is a lesson for another day.

Of course, there are always times to break the testing rules. Here is an example of one we ran for Intuit's QuickBooks Point of Sale Software. The wanted us to come up with a piece of creative they could run against another company's creative and their current control package. I'm not a fan of these types of horse races, because 1) I believe they are testing the least controlable factor (list and offer being first) and 2) They are not letting the agencies learn from the past.

After thinking about the proposal, we decided to change the rules. We knew the list was good and that most of the prospects were already using QuickBooks. We knew that once the prospect tried the software, it would be very difficult to switch. So instead of sending prospects an offer to request a trial CD, we sent the trial CD out in the first direct mail piece. Here is an example of that piece.