Experiments in Automated Messages Overview

Automated message experiments take advantage of machine learning to automatically favor better performing variants over time. This strategy is somewhat different than the traditional A|B split test, but accomplish the same objective with more efficiency and less work for you.

Automated Message Experiment Strategy

The experiment strategy for automated messages is called Multi Armed Bandit. This algorithmic strategy starts by giving each variant an equal opportunity to perform, and over time detects which variant is performing the best using machine learning. Once the algorithm learns which variant performs the best, it begins to favor that variant over the others.

Multiple experiments may be run concurrently in automated messages as the strategy (Multi Armed Bandit) is the same for a subject line experiment or body experiment.

Configuring an Automated Message Experiment

Publishing, or republishing a draft message is the mechanism to stop a test and to start a new one.

Below are the steps to set up an automated message experiment:

1. Set up a message with multiple variants (Ex: A,B,C...) to test. The variants can be a subject line test (configured via the UI) or in the message body using smarty code (see example below).
2. Publish the message and the system will begin by evenly distributing the variants each time it is sent via the trigger you have set up. Over time the algorithm will detect if one of the variants is outperforming the others and will begin to select and distribute that variant over the others. This strategy is called Multi Armed Bandit.
3. You can observe the results in the Experiments tab on the left side of the automation template UI.
   
   

Results for both subject line and body experiments can be exported into a CSV file. While on the Experiments page, click Export to download the results locally.

Subject Line Experiment

To set up a subject line experiment in an automated message, enable the experiment in the Message Header section, give the experiment and name and choose the conversion metric to optimize.

The following conversion available:

Below is an example of a subject line experiment testing 3 variants:

Body Experiment

Automation Body experiments are set up very similar to batch message body experiments, however there is no need to define the strategy.

The Experiment Element

The Variant Element

Example

The following example will test 3 variants with a conversion metric metric of unique clicks:

{experiment name="Test-Call-to-action" conversion="uniqueclicks"}
{variant name="TestA"} image and link A {/variant}
{variant name="TestB"} image and link B {/variant}
{variant name="TestC"} image and link C {/variant}
{/experiment}