Website Logo
Solutions
Teams
Marketers
Data Leaders
Industries
Finance
Retail
Sports
Other Industries
Integrations
Snowflake
Google BigQuery
Amazon Redshift
Data Warehouses
Destinations
Solutions
CRM Intelligence
Speak to your customers intelligently
Touchpoint Intelligence
See everything you say to customers
Cartographer
New
Manage UTM Parameters, get better reports
Pricing
Resources
BlogWebinarsWhitepapersDemos & VideosRecipesHelp Center
Company
AboutPressCareersSecurity
Get Demo
Phone menu toggle
  1. Help Center
  2. Campaign & Audience Evaluation

How-To: Setup a Treatment/Control Group for an Audience

How to setup a treatment/control split for an audience.
By 
Nolan Kruse
Last updated: 
January 26, 2022

In this article, we will walk through the steps to setup a treatment/control split for an experimental audience. If you are curious what this treatment/control split does, how to pick the appropriate split sizing, or tips for maximizing your chances of achieving statistical significance in your campaign evaluation, we highly encourage you to check out our Flywheel Campaign Evaluation Methodology Explained article.

‍

Step 1: Build your Audience

Creating your experimental audience is easy with Flywheel Audience Builder! Simply apply the Flex Filters necessary to narrow down your customer segment to just those fitting the desired criteria. Step-by-step instructions for how to do this can be found here if needed.

Step 2: Select your Treatment/Control Split

Before you save your audience, take a look at the Audience Report on the right hand side of the screen. Here you can see a live view of how many customers fit the criteria defined by the selected Flex Filters. You can also move the slider to adjust how many customers fall within the treatment group (who receive the marketing or sales intervention), and those who fall within the control group (who do NOT receive the intervention for this specific campaign). The default split shown is not necessarily a suggestion, but rather the default split established in the settings of your Audience Builder dataset, and can be changed at any time.

There are several trade-offs to keep in mind when selecting the Treatment / Control Split percentage:

There is often the temptation to bias the sample towards the treatment group in order to maximize the number of users receiving the marketing or sales intervention. Generally speaking, however, the best experimental comparisons between treatment and control groups will be achieved when the size of the groups is similar.

That said, the overall size of the audience matters a great deal when using statistical significance testing to evaluate the effectiveness of a campaign. Larger treatment group splits can be warranted if the overall size of the audience is suitably large.

When in doubt, Flywheel can perform what is called a Statistical Power Analysis - an analysis used to determine the minimum sample sizes and effect size (% incremental benefit in the treatment group) required to achieve statistical significance in an experiment.

Common Approaches to Treatment/Control Split:

When evaluating "evergreen" campaigns - campaigns that will be active on an ongoing basis and apply to any new users who qualify - many marketers will opt to split their campaign into two distinct phases.

In the first phase (the assessment phase), a robust control group will be used to ensure a proper determination of the campaign's effectiveness can be achieved.

In the second phase, assuming the assessment phase resulted in a sufficiently impactful incremental performance boost, the size of the control group will be reduced to ~20% (depending on audience size) to maximize the impact of the campaign while still retaining the ability to track the campaign's performance directionally.

‍

Step 3: Save your Audience and Begin Export to Destination(s)

Once you are happy with the treatment/control split, all that is left to do is to save your audience and begin exporting it to your desired marketing or sales destinations! Campaign Evaluation will kick off automatically and will analyze the behavior patterns of those customers who fall in the treatment group and compare to those in the control group.

‍

If you have further questions, please check out our Help Center or reach out at anytime to solutions@flywheelsoftware.com!

You can find our full list of subprocessors in our DPA.

Related Articles

How-To: View Revenue Uplift for your Audience
How to view revenue uplift in the Flywheel Audience Builder and understand if your campaign is working (or not).
Flywheel Campaign Evaluation Methodology Explained
An in-depth look into Campaign & Audience Evaluation with Flywheel

Get Personal Support

Leave a message
call anytime
1800 583 281
Flywheel Software Logo

The Easiest way to Create Customer Segments on your Data Warehouse

LinkedIn ProfileYoutube ChannelTwitter ProfileAngelList ProfileGitlab ProfileCrunchbase Profile
Company
  • Careers
  • About
  • Contact
  • Help Center
  • Service Desk
  • Blog
  • Security
Product
  • Destinations
  • Data Warehouses
  • Recipes
Solutions
  • For Marketers
  • For Data Teams
Policies
  • Terms of Service
  • Website Terms of Service
  • Vulnerability Disclosure Policy
  • Privacy
  • Data Processing Addendum
  • GCP - End User License Agreement
  • AWS - End User License Agreement
  • Bug Bounty Policy
© Flywheel Software 2022