Overview

Podium orchestration experiments provide a simple yet powerful way to create and test multiple automations against one another—with the ability to select the best performing automation.

Experiments can be used in a variety of ways. For example, you could set one trigger to launch an experiment with as many paths or variants you need. Or create an experiment after a customer has taken action. Another possibility is to create an experiment using different channels, like SMS versus email, to see which message performs better.

Contact your CSM to have orchestration experiments enabled in your account.

Features

• Add an experiment component anywhere within the orchestration
• View visual variant and control paths
• Set audience splits by percentage or static values
• View experiment analytics in the orchestration
• Select an experiment winner
• Supports Email, SMS, Push, and Open channel experiments

Add experiment

To get started, create a new Podium orchestration or open an existing orchestration. To add an experiment to the orchestration, hover over the Plus button and select Add experiment.

Experiment settings

In the Experiments pop-up, enter the name of your experiment, select how you want to split your audience, and add variants.

Split per variant or allow a control variant

The Experiments settings pop-up gives you the ability to split the audience evenly per variant or allow a control variant.

• Splitting per variant will divide your audience evenly across the number of variants you have created. For example, if you split the variant evenly across four variants, 25% will receive variant A, 25% will receive variant B, 25% will receive variant C, and 25% will receive variant D.
• Allowing for a control variant will divide your audience either by Percentage of total audience or a specified Number of contacts.

In order to use a control variant in a Podium experiment, the experiment must be built out at the Podium level. Message body experiments in Podium use the multi-armed bandit strategy, which doesn't allow for control variants.

The control variant isn’t part of the experiment and will send messages at the same time as Variant A and Variant B.

If Percentage of total audience is selected, a slider bar will appear, and you can set the percentage of the audience that receives the variants and the control. For example, if you have two variants and a control, and you indicate that 50% of the tests will go to the variants and the other 50% will go to the control, then 25% will go to Variant A and 25% to B, with the remaining going to the control variant when the experiment is enabled.

If Number of contacts is selected, an input field will display where you can set the number of contacts that receive each variant. For example, if you have two variants and a control, and you indicate that 100 contacts should receive the variant, the tests will go to the 100 variants and the remaining contacts will receive the control variant.

Once your experiment settings are complete and you click the continue button, you will see the different variants and/or control variants in the orchestration. The Delay, Filter, and Action components can be created.

Publish and enable

Once all components are configured and your message content is created, you can publish the entire orchestration by clicking the Publish Draft button.

If any components are left incomplete, you will see an error with the incomplete components highlighted.

After successfully publishing the orchestration, you can enable it using the Enable/Disable dropdown. In the published state, the delay, filter, and action components are read-only until the orchestration is copied as a draft.

When an orchestration is running and then disabled, all message sending will be stopped for that orchestration, even if the contact is in the delayed state. Once the orchestration is enabled again, contacts will continue on the path from their last state, and all updates to Podium insights will resume.

View experiment analytics

When an orchestration is published and enabled and the trigger is initiated, orchestration analytics will display next to each variant on the Published tab. The following metrics are available:

• Delayed: the number of triggered contacts in the delayed state.
• Filtered Out: the number of contacts filtered out of the automation.
• Email Sent: the number of contacts that received the email message.
• SMS Sent: the number of contacts that received the SMS message (if SMS is enabled in the account).
• Push Sent: the number of contacts that received the push message (if Push is enabled in the account).
• Message Stopped: the number of contacts that met the Message Stopped criteria.

When a draft orchestration containing one or more experiments is republished, all Podium Insights will be reset.

It's possible for a child automation within an orchestration to have more trigger contacts than are displayed in Total Triggered. The reason is that when you republish, the delayed contacts move over to the new version of the orchestration. You need to look at past and published stats for a holistic view.

Select an experiment winner

Once you’ve determined which automation within the experiment performs best, you can select a winner and start using it in a new draft. To select the winner, click the Copy to Draft button and hover over the Experiment component. The Select Winner drop-down will display. Click on Select Winner, and a new draft will be created.

Once a winner is selected and the orchestration republished, the analytics will restart.