Overview
A subject line experiment allows you to test multiple subject lines in a single message to optimize the following conversion metrics:
Conversion Metric | Description |
---|---|
Unique Opens | The metric is calculated by measuring the total unique opens for the message. Ideal for testing subject lines. |
Unique Clicks | The metric is calculated by measuring the total click to open rate for the message. |
Revenue to Sent | The metric is calculated by dividing the total revenue attributed to the message by the total number of messages sent. |
Order to Sent | The metric is calculated by dividing the total number of orders attributed to the message by the total number of messages sent. |
Revenue to Click | The metric is calculated by dividing the total revenue attributed to the message by the total number of unique clicks. |
Order to Click | The metric is calculated by dividing the total number of orders attributed to the message by the total number of unique clicks. |
Note: Future enhancements will include other KPIs such as custom conversion events.
There are two experiment options available:
- Split test
- Split your audience evenly per variant.
- Set the variant sample size (in percentage or number) and allow for a control variant.
- Split test with champion: Set sample size per variant (in percentage or number) and automatically send a champion.
Configure a subject line experiment
Create a new message or open a draft and edit the message header.
Enable the experiment
Enable the subject line experiment with the toggle switch.

Name the experiment
This is the overall name of your experiment. Names can only contain alphanumeric characters, dashes, underscores and no more than 30 characters (i.e. Holiday_Subject_Test).

Choose the conversion metric
Choose what you’d like to optimize for. This is the metric that will be used to determine which variant performs better.
You're able to optimize based on a contact's message engagement (unique opens and unique clicks), orders (orders to open and orders to click), or revenue (revenue to open and revenue to click).

Select the experiment strategy
Choose either a Spilt Test or a Split Test with Champion.
Split test
A split test will determine how the audience is split for the experiment.
- Split audience evenly per variant: splits the audience evenly between the variants.
- Allow a control variant: allows you to add a control variant in addition to the experiment variant(s).
- You can choose the split based on a percentage or a number.

Allowing for a control variant lets you set aside a group of people (control) to test your variant against.
You will see the control variant input (light green box) show up below the variants.
You are able to set the test size as a percentage or a specific number of contacts.

Split test with champion
When the strategy is set for split test with champion, you are able to have the remainder of the audience be sent to the best performing variant after a specified delay.
You may set the test size as a percentage or a specific number.

You have now configured your experiment.
Next, configure the variants.
Configure variants
Name both of your variants (note: only alphanumeric characters, dashes, underscores).
Enter the subject lines to test.
Click + Subject Variant to add more variants to test.

Save
Once finished, click Save.
You will see a preview of the details in the message header.
Your subject line experiment has now been configured.

Subject line experiment scenarios
Split test without champion
Test n variants where all variants are evenly split across the audience.
- 50% A
- 50% B

Test n variants where all variants are evenly split across the audience.
- 33.3% A
- 33.3% B
- 33.3% C

Test n test variants (same size population) and 1 control as remainder.
- 1200 Variant A
- 1200 Variant B
- Remainder control

Test n test variants (as a percentage) and 1 control as remainder.
- 40% Variant A
- 40% Variant B
- Remainder control (20%)

Test 1 single variant and 1 control as remainder.
- 1500 Variant A
- Remainder control

Split test with champion
Test n variants and auto-send champion after x mins/hours.
- 500 sent A
- 500 sent B
- Remainder sent to the champion after 3 hours.

Test n variants and auto-send champion after x mins/hours.
- 20% sent A
- 20% sent B
- Remainder sent to the champion after 3 hours (60%).

In the next article, you can learn about body experiments.
Comments
0 comments
Please sign in to leave a comment.