Overview

Automated message experiments take advantage of machine learning to automatically favor better performing variants over time. This strategy is somewhat different than the traditional A|B split test, but it accomplishes the same objective with more efficiency and less work for you.

Automated message experiment strategy

The experiment strategy for automated messages is called multi-armed bandit. This algorithmic strategy starts by giving each variant an equal opportunity to perform, and over time detects which variant is performing the best using machine learning. Once the algorithm learns which variant performs the best, it begins to favor that variant over the others.

Multiple experiments may be run concurrently in automated messages as the strategy (multi-armed bandit) is the same for a subject line experiment or body experiment.

The multi-armed bandit strategy doesn't allow for control variants.

Create an automated message experiment

Publishing or republishing a draft message is the mechanism to stop a test and to start a new one. Below are the steps to set up an automated message experiment.

1. Set up a message with multiple variants to test. The variants can be a subject line test in the UI—Message Automations > Create New Automation > edit Message Header—or in the message body using Smarty code (see example below).

2. Publish the message and the system will begin by evenly distributing the variants each time it's sent via the trigger you have set up. Over time the algorithm will detect if one of the variants is outperforming the others and will begin to select and distribute that variant over the others.

3. You can observe the results by clicking the Experiments tab on the left side of the automated message creation page.

4. Results for both subject line and body experiments can be exported into a CSV file. While on the Experiments page, click Export to download the results locally.

Subject line experiment

1. To set up a subject line experiment in an automated message, navigate to Message Automations > Create New Message or edit an existing draft.

2. Click Edit in the Message Header pane and switch the experiment toggle to On.

3. Give the experiment a name and choose the conversion metric in the Optimize dropdown.

The following conversion metrics are available:

To take advantage of order and revenue conversion metrics, you need to be passing order and revenue data to your account via the API or JavaScript listeners.

Example of a subject line experiment testing three variants

AI-generated subject lines

You can utilize AI-generated experiment subject lines to optimize the impact of your messaging. Generative AI is available for experiment variants so you can analyze how people react to different subject lines.

1. Click Generate AI Content in the Subject experiment pane.

3. From within the generative AI pane, enter a prompt and click Generate AI Content. Multiple variations of usable content will appear.

4. To customize the text, select one or multiple of the buttons: Shorten, Lengthen, Urgent, Casual, Funny.

To keep your subject lines concise and effective, AI content in this pane won't exceed one line.

5. Click Use Content under the AI-generated text you wish to use, and it will appear in your message subject line.

Body experiment

Automation body experiments are set up in a very similar way to batch message body experiments, but there's no need to define the strategy.

Experiment element

Variant element

Example

The following example will test three variants with a conversion metric of unique clicks.

{experiment name="Test-Call-to-action" conversion="uniqueclicks"}
{variant name="TestA"} image and link A {/variant}
{variant name="TestB"} image and link B {/variant}
{variant name="TestC"} image and link C {/variant}
{/experiment}