Overview

Full Message Experiments help you understand what’s truly working in your batch email campaigns, by streamlining the setup, launch and analysis of variants and automatically measuring statistically meaningful results.

Full Message Experiments can be created in two ways:

1. A guided process that creates, targets, launches, and analyzes a new experiment
2. Comparative analysis of existing sent messages

Creating a new Full Message Experiment allows rapid content entry, audience selection and scheduling of a test which will be launched, collected and generate a report. It also allows for additional test launch within a single experiment and aggregation of overall performance under a single hypothesis.

Creating a Full Message Experiment report quickly generates results for any previously sent batch messages that use Cordial’s Random Number Generator (RNG) method for audience splits. This lets you easily analyze performance and identify winning variants without manual spreadsheets or offline analysis. 

Key benefits of both methods include:

• Easy setup steps: rapidly and repeatably draft, target and launch
• Automatic, locked audience splits: Removing the need for manual contact updates and segmentation
• Automatic, immediate results: winner declaration and confidence scoring (clicks, orders, RPE, opt outs)
• Higher accuracy: Results aggregate across messages to improve statistical reliability.
• Faster learning: Quickly identify which messages and AI features drive stronger engagement or revenue.

What Are Full Message Experiments?

Full Message Experiments are designed to help you run and analyze experiments across entire messages, not just subject lines or single components. They include several key layers of organization and visibility.

1. Full Message Experiments are the container that groups together multiple batch messages and provides an overall performance summary.
2. Test Groups are the sets of tests that include variants from a single send instance. These are commonly the Control and Challenger variants of a send on a single day. Multiple test groups can be added to one experiment.
3. Messages are the variants within each test group that use RNG-split audiences and are intended to be grouped together for analysis.

Examples

Create a new Experiment with a non-AI equipped control and a challenger variant that uses AI-driven product recommendations. Designate a high engagement audience that may be receptive to recommendations. Launch the test and watch the performance page declare a winner. Export for extended analysis. Launch a second test with the same split audience and new content variance, compare the automatic test and aggregate results. Use the winning strategy across all future campaigns.

Use Experiment Reporting to track an ongoing test of AI-driven product recommendations across five batch email sends. Add each send to the experiment, designate your control and test variants, and view aggregated results, including click rate, order rate, and revenue per email, to determine whether the test variant is producing a statistically significant lift over time.

Before you begin

Before using Full Message Experiments, confirm the following:

• Your account has Full Message Experiments enabled. This feature is available to select accounts. Contact your Client Success Manager or submit a support request for access.
• If using the existing message analysis feature, ensure you have already sent the batch email messages you want to analyze. Messages analyzed with this method should be using Cordial's prior random number generation method to split your audience into control and test groups.

Creating a New Full Message Experiment

Launching a new experiment follows a streamlined 3 step process.

1. Navigate to Analytics -> Full Message Experiments
2. Select New Experiment
3. Select Add New Messages

4. Enter basic details about your new experiment and choose the primary test metric that you will declare a winner against.

5. Add the Audience that all tests within this experiment will run against. This audience will be automatically split into test groups. Optionally update the non-Control variant name.

6. Assign a name, tags, editor type and priority to the first message of the experiment

8. Schedule, Audience, Header and Content editing are now available for the message. These may be populated according to the desired test strategy.

9. The Experiment Overview page will allow updates to the Description and Primary test metrics. The Audience may be edited up until the first test is launched/sent.

10. Once launched, the Performance page will automatically reflect delivery statistics, metric analysis and winner declaration. Tooltips will convey statistical significance per metric and all statistics may be exported.

11. To launch additional tests within this same experiment, navigate to Test Groups->Add Test Group. This will produce a Create New Message dialog and make the second message available for editing. All messages within a single experiment will share the same locked audience split.

Note: Once launched or scheduled, messages will show under the Email Sent or Scheduled pages and within all account-wide reports.

Creating a Full Message Experiment Report

Adding RNG‑enabled batch emails to an experiment report and generating results is straightforward.

1. Navigate to Analytics -> Full Message Experiments
2. Select New Experiment
3. Select Add Existing Messages
    

3. Name your experiment, add a description and select the Primary test metric. 
    

   

    

4. Enter a new label for your test variants or use the default labels of Control and Challenger
    

5. Create your first test group and select the batch messages that correspond to each. Optionally, add multiple test groups and sets of messages.
    

6. An aggregate performance report is immediately generated.

Review experiment results

After adding your messages, the experiment performance page aggregates results across all message groups and displays the following:

• Key metrics: open rate, click rate, order rate, revenue per email, unsubscribe rate, bounce rate, and delivered rate, shown per variant.
• Relative lift: the performance difference between your test and control variants across each metric.
• Statistical winner: which variant is performing better, based on the volume and results of the sends included in the experiment.
   

The accuracy and reliability of your results depend on how your audience splits were set up prior to sending. Experiment Reporting analyzes the data from your existing sends, it does not validate or enforce experiment methodology during setup.