Embarrassing A/B Test failure - what went wrong?

Hi All,

Just spent the past week and a half designing and testing an email, and was especially pleased to have it selected for the Campaign Monitor design gallery! Woot!

However, when it came to sending it we had a royal goofup.

I did an A/B test on the subject line - altering the end of the B to include some more product info. Both were personalised with exactly the same fields: I copied and pasted from A to B, having thoroughly tested A, which worked flawlessly every time.

The A/B went to 4000 people, and after 3 hours B had won and duly went out to 52,000 subscribers.

How horrified I was to receive an email in my inbox with the subject line: [FirstName1,Fallback=Customer] [LastName1,Fallback=].

I spent the rest of the afternoon replying to disgruntled subscribers, and generally feeling very embarrassed.

What went wrong?

Jim

travisbell travisbell, 7 years ago

Hey Jim,

Sorry about this. I believe I just grabbed your ticket from the support inbox and once I get a hold of your HTML I'll be able to troubleshoot this further for you.

Thanks and sorry for the trouble!


--
Travis Bell
travisbell travisbell, 7 years ago

Hey Jim,

Just wanted to update the forum too, in case anyone else happens to have seen this issue.

The short and simple version of what happened is that this was our fault, due to a bug in the A/B testing area. We have the devs on it and it will be fixed shortly. If anyone else saw this problem in the past week or two, make sure to get in touch with us.

Sorry about the trouble and for embarrassing you! Certainly not our intent! If you have any further questions or concerns feel free to let me know.


--
Travis Bell

Join 200,000 companies around the world that use Campaign Monitor to run email marketing campaigns that deliver results for their business.

Get started for free
1-888-533-8098