Automated message experiments take advantage of machine learning to automatically favor better performing variants over time. This strategy is somewhat different than the traditional A|B split test, but accomplish the same objective with more efficiency and less work for you.
Automated Message Experiment Strategy
The experiment strategy for automated messages is called Multi Armed Bandit. This algorithmic strategy starts by giving each variant an equal opportunity to perform, and over time detects which variant is performing the best using machine learning. Once the algorithm learns which variant performs the best, it begins to favor that variant over the others.
Multiple experiments may be run concurrently in automated messages as the strategy (Multi Armed Bandit) is the same for a subject line experiment or body experiment.
Configuring an Automated Message Experiment
Publishing, or republishing a draft message is the mechanism to stop a test and to start a new one.
Below are the steps to set up an automated message experiment:
- Set up a message with multiple variants (Ex: A,B,C...) to test. The variants can be a subject line test (configured via the UI) or in the message body using smarty code (see example below).
- Publish the message and the system will begin by evenly distributing the variants each time it is sent via the trigger you have set up. Over time the algorithm will detect if one of the variants is outperforming the others and will begin to select and distribute that variant over the others. This strategy is called Multi Armed Bandit.
- You can observe the results in the Experiments tab on the left side of the automation template UI.
Results for both subject line and body experiments can be exported into a CSV file. While on the Experiments page, click Export to download the results locally.

Subject Line Experiment
To set up a subject line experiment in an automated message, enable the experiment in the Message Header section, give the experiment and name and choose the conversion metric to optimize.
The following conversions are available:
Conversion Metric | Description |
---|---|
Unique Opens | The metric is calculated by measuring the total unique opens for the message. Ideal for testing subject lines. |
Unique Clicks | The metric is calculated by measuring the total click to open rate for the message. |
Revenue to Sent | The metric is calculated by dividing the total revenue attributed to the message by the total number of messages sent. |
Order to Sent | The metric is calculated by dividing the total number of orders attributed to the message by the total number of messages sent. |
Revenue to Click | The metric is calculated by dividing the total revenue attributed to the message by the total number of unique clicks. |
Order to Click | The metric is calculated by dividing the total number of orders attributed to the message by the total number of unique clicks. |
Complaints Rate | This metric is calculated by dividing the total number of complaints by the total number of messages sent. |
Opt Out Rate | This metric is calculated by dividing the total number of opt out events by the total number of messages sent. |
Below is an example of a subject line experiment testing 3 variants:

Body Experiment
Automation Body experiments are set up very similar to batch message body experiments, however there is no need to define the strategy.
The Experiment Element
Attribute | Expected | Required |
---|---|---|
name | string | true |
conversion | uniqueclicks uniqueopens totalrevenue/totalsent totalorders/totalsent totalorders/uniqueclicks totalrevenue/uniqueclicks |
true |
The Variant Element
Attribute | Expected | Required |
---|---|---|
name | string | true |
Example
The following example will test 3 variants with a conversion metric of unique clicks:
{experiment name="Test-Call-to-action" conversion="uniqueclicks"}
{variant name="TestA"} image and link A {/variant}
{variant name="TestB"} image and link B {/variant}
{variant name="TestC"} image and link C {/variant}
{/experiment}
Comments
0 comments
Please sign in to leave a comment.