Subject Line Experiments

Overview

A subject line experiment allows you to test multiple subject lines in a single message to optimize the following conversion metrics:

Note: Future enhancements will include other KPIs such as custom conversion events.

There are two experiment options available:

• Split test
  • Split your audience evenly per variant.
  • Set the variant sample size (in percentage or number) and allow for a control variant.
• Split test with champion - Set sample size per variant (in percentage or number) and automatically send a champion.

Configuring a Subject Line Experiment

Create a new message or open a draft and edit the message header.

Enable the Experiment

Enable the subject line experiment with the toggle switch.

Name the Experiment

This is the overall name of your experiment.  Names can only contain alphanumeric characters, dashes, underscores and no more than 30 characters (i.e. Holiday_Subject_Test).

Choose the Conversion Metric

Choose what you’d like to optimize for. This is the metric that will be used to determine which variant performs better.

You're able to optimize based on a contact's message engagement (unique opens and unique clicks), orders (orders to open and orders to click), or revenue (revenue to open and revenue to click). 

Select the Experiment Strategy

Choose either a Spilt Test or a Split Test with Champion.

Split Test

A split test will determine how the audience is split for the experiment.

• Split audience evenly per variant - splits the audience evenly between the variants.
• Allow a control variant - allows you to add a control variant in addition to the experiment variant(s).
  • You can choose the split based on a percentage or a number.

Allowing for a control variant lets you set aside a group of people (control) to test your variant against.

You will see the control variant input (light green box) show up below the variants.

You are able to set the test size as a percentage or a specific number of contacts.

Split Test with Champion

When the strategy is set for split test with champion, you are able to have the remainder of the audience be sent to the best performing variant after a specified delay.

You may set the test size as a percentage or a specific number.

You have now configured your experiment.

Next, configure the variants.

Configure Variants

Name both of your variants (note: only alphanumeric characters, dashes, underscores).

Enter the subject lines to test.

Click + Subject Variant to add more variants to test.

Save

Once finished, click Save.

You will see a preview of the details in the message header.

Your subject line experiment has now been configured.

 

Subject Line Experiment Scenarios

Split test without champion

Test n variants where all variants are evenly split across the audience.

• 50% A
• 50% B

Test n variants where all variants are evenly split across the audience.

• 33.3% A
• 33.3% B
• 33.3% C

Test n test variants (same size population) and 1 control as remainder.

• 1200 Variant A
• 1200 Variant B
• Remainder control

Test n test variants (as a percentage) and 1 control as remainder.

• 40% Variant A
• 40% Variant B
• Remainder control (20%)

Test 1 single variant and 1 control as remainder.

• 1500 Variant A
• Remainder control

Split test with champion

Test n variants and auto-send champion after x mins/hours.

• 500 sent A
• 500 sent B
• Remainder sent to the champion after 3 hours.

Test n variants and auto-send champion after x mins/hours.

• 20% sent A
• 20% sent B
• Remainder sent to the champion after 3 hours (60%).

In the next article learn about body experiments.