Marketing Automation allows you to create A/B Testing campaigns, in which you can test different versions of the same message to samples of recipients to see which is more effective. After you configure and deploy the campaign, Marketing Automation will manage sending each version and tracking the results. At a date and time you set, it will determine which version is the winner of the test according to the criteria you entered during campaign setup.

Starting a New A/B Testing Campaign

Set up an A/B Testing Campaign

Select Campaign Members

Setup Test Variable

Determine Winner

Review and Deploy an A/B Testing Campaign

.

Starting a New A/B Testing Campaign

To use Marketing Automation’s A/B testing feature, select ‘A/B Testing’ for campaign type when you create a new campaign.

.

Set up an A/B Testing Campaign

Once you have selected an A/B campaign type, you will go through 3 pages of of campaign creation. On the first page is where you will configure the campaign.
.

1. Select Campaign Members

The first choice to make is which recipients will be enrolled.

Enrollment is restricted to groups for A/B Testing campaigns. Click the Select Group(s) link to choose one or more groups.

Note: Only recipients who are in the group when the campaign is deployed will be enrolled in the campaign. If a recipient is added to a group after the campaign is deployed, they will not be enrolled.

Second, select how many samples you would like to use in your campaign. You can have anywhere from 2 to 26. Click Submit to create the samples.

Third, choose how large each sample should be. You can choose to set the sample size based on the number of recipients or a percentage of the total.

  • If you choose recipients in the drop-down, the number you enter in the text field will be the number of recipients added to each sample when the campaign is deployed.

Note: Because recipients are not enrolled until the campaign is deployed, it’s possible to have a larger number of recipients assigned to the samples than are in the campaign. If you want to use large sample sizes, you may want to first conduct a Marketing Automation Recipient Search for the groups you are using for enrollment. This will give you an estimate of how many recipients you can expect.

  • If you choose % of total in the drop-down, the number you enter in the text field will be the percentage of recipients added to each sample when the campaign is deployed. The value must be between 1 and 100. A slider will also appear below the text field. Dragging the slider is another way to set the percentages.

.

2. Setup Test Variable

Select the variable you would like to test in the drop down.  The available choices are messagesubject linepre-headerfrom field, and time of send.

  • Select message if you would like  to send a completely different message to each group. You will be prompted to select the message to send to each group. You will then enter a send date and time. Click the send options link to enter the other information, such as subject line, from address, etc., which apply to all of the messages sent.
  • If you select subject line, you will be prompted to enter the subject line for each sample. You will then enter a send date and time. Click the send options to select the message and enter the other information which apply to all of the messages sent.
  • Selecting from address works similarly to selecting subject line. You will be prompted to enter a from address for each sample. If your account has a default from address option, it will appear. Enter the send date and time, and then click the send options link to select the message as well as the other information which apply to all of the messages sent.
  • Select time of send to have each sample message sent at a different time. Enter the date time you would like each message to be sent. Click the send options to select a message and enter the other send information.

.

3. Determine Winner

Specify how Marketing Automation should determine which message “wins” the A/B test. You can choose to for the winner to be based on open rate or click rate. Open rate is the percentage of messages opened by the recipients in each sample; click rate is the number of clicks per message in each sample.

You can choose either to have no minimum rate or a minimum rate for the winner.

  • If you choose no minimum rate, then whichever sample has the better percentage will become the winner.
  • If you choose a minimum rate, you will be prompted to enter a percentage. In order to be considered the winner, a sample has to have an open/click rate equal to or greater than the percentage you enter.
    • If no sample meets that threshold, then no sample will be considered the winner.

Enter a date and time to determine the winner.

  • This is the time that Marketing Automation calculates which message “won.”
  • It must be after the last send time for allmessages sent.
  • It is a best practice to give your samples plenty of time to determine a winner. Typically, this is one or two business days. Less time will prevent your recipients from all having a chance to see the message.

There are two options you can choose from when a winner is determined.

  • The first option is to send a notification when the winner is chosen. The notification will be sent to the email you enter with the results of the A/B test.
  • The second option is to automatically send the winner to the remaining campaign members. If you select this option, then after the winner is determined, it will be sent to recipients in the campaign who were not sent any test message.

Example: You run a test with three messages – A, B, and C – which are each sent to 10% of the enrolled recipients. When the test is finished, message A has won. Any recipient who was not sent any test message will be sent message A. Any recipient who was sent messages B or C will not be sent message A.

If the A/B test was for time of send, you will need to specify what day the message will be sent.

  • For either option, the message will be sent at the time which “won” the A/B test.
  • Choose on the next day to have the message sent on the next business day (e.g., if the winner is determined on a Tuesday, then the message will be sent on Wednesday.)
  • Choose on the same day of the week to wait until the original day of the week cycles back (e.g., if you sent the message on a Tuesday, it will wait until the next Tuesday to send it.)

If you do not choose to send the winner automatically, you will need to return to this page after the winner is determined. Enter the date and time you would like the winner to be sent, and then click the Send remainder button to schedule the send action. When you do, the winning message will be sent to recipients in the campaign who were not sent any test message.

Example: You run a test with three messages – A, B, and C – which are each sent to 10% of the enrolled recipients. When the test is finished, message A has won. Any recipient who was not sent any test message will be sent message A. Any recipient who was sent messages B or C will not be sent message A.

.

Review and Deploy an A/B Testing Campaign

Click Next to go to the page where you review the campaign configuration. 

Click Next again to go to the ‘Deploy Campaign’ page. On this page, you can schedule the campaign to start or end on a specific date. Click Deploy to enroll recipients and begin your A/B Test.