How can we help?

Subject Line Experiments

Overview

A subject line experiment allows you to test multiple subject lines in a single message to optimize the following conversion metrics:

Conversion Metric Description
Unique Opens The metric is calculated by measuring the total unique opens for the message. Ideal for testing subject lines.
Unique Clicks The metric is calculated by measuring the total click to open rate for the message.
Revenue to Sent The metric is calculated by dividing the total revenue attributed to the message by the total number of messages sent.
Order to Sent The metric is calculated by dividing the total number of orders attributed to the message by the total number of messages sent.
Revenue to Click The metric is calculated by dividing the total revenue attributed to the message by the total number of unique clicks.
Order to Click The metric is calculated by dividing the total number of orders attributed to the message by the total number of unique clicks.

Note: Future enhancements will include other KPIs such as custom conversion events.

There are two experiment options available:

  • Split test
    • Split your audience evenly per variant.
    • Set the variant sample size (in percentage or number) and allow for a control variant.
  • Split test with champion: Set sample size per variant (in percentage or number) and automatically send a champion.

Configure a subject line experiment

Create a new message or open a draft and edit the message header.

Enable the experiment

Enable the subject line experiment with the toggle switch.

Name the experiment

This is the overall name of your experiment.  Names can only contain alphanumeric characters, dashes, underscores and no more than 30 characters (i.e. Holiday_Subject_Test).

Choose the conversion metric

Choose what you’d like to optimize for. This is the metric that will be used to determine which variant performs better.

You're able to optimize based on a contact's message engagement (unique opens and unique clicks), orders (orders to open and orders to click), or revenue (revenue to open and revenue to click). 

Note: To take advantage of order and revenue conversion metrics, you need to be passing order and revenue data to your account via the API or JavaScript listeners.

Select the experiment strategy

Choose either a Spilt Test or a Split Test with Champion.

Split test

split test will determine how the audience is split for the experiment.

  • Split audience evenly per variant: splits the audience evenly between the variants.
  • Allow a control variant: allows you to add a control variant in addition to the experiment variant(s).
    • You can choose the split based on a percentage or a number.

Allowing for a control variant lets you set aside a group of people (control) to test your variant against.

You will see the control variant input (light green box) show up below the variants.

You are able to set the test size as a percentage or a specific number of contacts.

Split test with champion

When the strategy is set for split test with champion, you are able to have the remainder of the audience be sent to the best performing variant after a specified delay.

You may set the test size as a percentage or a specific number.

You have now configured your experiment.

Next, configure the variants.

Configure variants

Name both of your variants (note: only alphanumeric characters, dashes, underscores).

Enter the subject lines to test.

Click + Subject Variant to add more variants to test.

Save

Once finished, click Save.

You will see a preview of the details in the message header.

Your subject line experiment has now been configured.

 

Subject line experiment scenarios

Split test without champion

Test n variants where all variants are evenly split across the audience.

    • 50% A
    • 50% B

Test n variants where all variants are evenly split across the audience.

    • 33.3% A
    • 33.3% B
    • 33.3% C

Test n test variants (same size population) and 1 control as remainder.

    • 1200 Variant A
    • 1200 Variant B
    • Remainder control

Test n test variants (as a percentage) and 1 control as remainder.

    • 40% Variant A
    • 40% Variant B
    • Remainder control (20%)

Test 1 single variant and 1 control as remainder.

    • 1500 Variant A
    • Remainder control

Split test with champion

Note: Not currently supported with messages scheduled to time zone of the recipient.

Test n variants and auto-send champion after x mins/hours.

    • 500 sent A
    • 500 sent B
    • Remainder sent to the champion after 3 hours.

Test n variants and auto-send champion after x mins/hours.

    • 20% sent A
    • 20% sent B
    • Remainder sent to the champion after 3 hours (60%).

 

In the next article, you can learn about body experiments.

Comments

0 comments

Please sign in to leave a comment.