Overview

Podium Orchestration experiments provide a simple yet powerful way to create and test multiple automations against one another—with the ability to select the best performing automation so you can continue sending it out.

Features

• Add an experiment component anywhere within an Orchestration
• Set one trigger to launch an experiment with as many paths or variants as you need
• Create an experiment after a customer has taken action
• Easily A/B test messages across channels (like SMS versus email) to find what resonates with people
• View real-time analytics at a glance for each variant
• Set audience splits by percentage or static values
• Select an experiment winner to continue sending
• Experiments are available for Email, SMS, Push (mobile), and the REST channel

Add experiment

1. To get started, navigate to Automations > Podium Orchestrations and create a new Podium Orchestration or open an existing Orchestration.

2. To add an experiment to the Orchestration, hover over the Plus button and select Add experiment.

Experiment settings

In the Experiments pop-up, enter the name of your experiment, select how you want to split your audience, and add variants.

Split per variant or allow a control variant

The Experiments settings pop-up gives you the ability to split the audience evenly per variant or allow a control variant.

• Splitting per variant will divide your audience evenly across the number of variants you have created. For example, if you split the variant evenly across four variants, 25% will receive variant A, 25% will receive variant B, 25% will receive variant C, and 25% will receive variant D.
• Allowing for a control variant will divide your audience either by Percentage of total audience or a specified Number of contacts.

In order to use a control variant in a Podium experiment, the experiment must be built out at the Podium level. Message body experiments in Podium use the multi-armed bandit strategy, which doesn't allow for control variants.

The control variant isn’t part of the experiment and will send messages at the same time as Variant A and Variant B.

If Percentage of total audience is selected, a slider bar will appear, and you can set the percentage of the audience that receives the variants and the control. For example, if you have two variants and a control, and you indicate that 50% of the tests will go to the variants and the other 50% will go to the control, then 25% will go to Variant A and 25% to B, with the remaining going to the control variant when the experiment is enabled.

If Number of contacts is selected, an input field will display where you can set the number of contacts that receive each variant. For example, if you have two variants and a control, and you indicate that 100 contacts should receive the variant, the tests will go to the 100 variants and the remaining contacts will receive the control variant.

Once your experiment settings are complete and you click the continue button, you will see the different variants and/or control variants in the Orchestration. The Delay, Filter, and Action components can be created.

A/B testing

1. To A/B test two or more variants against each other, create the desired variants, and configure each individual automation:

• Define the trigger
• Set the delay
• Set the audience
• Define the action

2. Once you've created and configured your variants, Publish and Enable the Orchestration to begin sending out the variants.

Possible use cases

• Shown above: send the same automated message across different channels to see which channel people interact with more for that particular message.
• Create two or more versions of possible content for a welcome or cart abandon series, see which content performs best, then select the winning variant to continue sending to future customers.

3. Once your Orchestration has started sending, go to the at-a-glance analytics next to each variant to see which one performs best.

4. If desired, select the best-performing variant as the experiment winner. This will copy the variant as a new draft of the Orchestration so you can continue sending the winning variant (without the need for additional configuration).

Publish and enable

Once all the components are configured and your message content is created, publish the entire Orchestration by clicking the Publish Draft button.

If any components are left incomplete, you will see an error with the incomplete components highlighted.

After successfully publishing the Orchestration, you can enable it using the Enable/Disable dropdown. In the published state, the Delay, Filter, and Action components are read-only until the Orchestration is copied as a draft.

When an Orchestration is running and then disabled, all message sending will be stopped for that Orchestration, even if the contact is in the delayed state. Once the Orchestration is enabled again, contacts will continue on the path from their last state, and all updates to real-time Podium analytics will resume.

Real-time analytics

1. When an Orchestration is published and enabled and the trigger is initiated, hover over an experiment variant on the Published tab to view analytics information.

2. The following high-level metrics are available:

• Delayed: number of triggered contacts in the delayed state
• Filtered Out: number of contacts filtered out of the automation
• Email Sent: number of contacts that received the email message
• SMS Sent: number of contacts that received the SMS message
• Push Sent: number of contacts that received the push message
• Message Stopped: number of contacts that met the Message Stopped criteria

3. Hover over Email Sent, SMS Sent, or Push Sent to view more detailed analytics:

• Sent: number of sent messages
• Delivered: number of sent messages that were successfully delivered
• Opened: number of sent messages that were opened
• Opened (adjusted): number of opened messages adjusted to remove bot opens (as detected by Cordial)
• Clicked: number of links within the message that were clicked
• Bounced: number of messages that bounced
• Opt-outs: number of messages that resulted in opt-outs
• Complaints: number of messages that resulted in complaints
• Revenue: revenue attached to message activity

When a draft Orchestration containing one or more experiments is republished, all real-time analytics will be reset.

It's possible for a child automation within an Orchestration to have more trigger contacts than are displayed in Total Triggered. The reason is that when you republish, the delayed contacts move over to the new version of the Orchestration. You need to look at past and published stats for a holistic view.

Select an experiment winner

Once you’ve determined which automation within the experiment performs best, you can select a winner and start using it in a new draft.

1. To select the winner, click the Copy to Draft button and hover over the Experiment component.

2. Click on Select a winner.

3. A new draft of the Orchestration containing only the experiment winner will be created.

4. Republish the experiment winner Orchestration to continue sending it out to applicable contacts.

Once a winner is selected and the Orchestration republished, the analytics will restart.