Join our Community in its new Home - The Datorama Trailblazer Community Group!

It's been an amazing 3 years coming together in this forum to collaborate, innovate, support, and inspire each other about our shared usage of Datorama. While this is not quite a goodbye, we are excited to announce that we are getting a fresh start in our new home within the Salesforce Trailblazer Community. We have a ton of fun new content planned and you may even see the revival of some of our most popular posts from the past few years.

We’ll be keeping this group around for a bit for you to peruse, but as of November 15, we will no longer be allowing new posts or comments. Be sure to join our new group at https://www.salesforce.com/success-datorama to keep the conversation going.

We can’t wait to see you there!

How to Implement A/B Split Testing in Datorama

_ben_easley_ben_easley SFOSYS_ADMIN IMG42
edited May 2019 in Show & Tell

Part 1 of 2


A couple of notes about this solution:
 
The following solution is applicable to the media buy granularity.  If your test is at a different grain (like the campaign or creative grain), you’ll need to adjust the granularity setting within your calculated metrics for the calculations to work as expected.  Additionally, the solution assumes you will have a single control media buy (or campaign, creative) per test group, but you may have any number of test media buys per test group.


→  See the A/B Split Test Demo Dash HERE


Getting Started
Ever since your agency adopted Datorama, you’ve been killing it as the Datorama admin. Your client services and sales teams are over the moon about the dashboards you’ve created, and your ad ops team is really loving the automated reporting you’ve put in place to help them with optimization and campaign monitoring.  Your team is relying on you to keep the good times rolling, and because Datorama is so much more than your run-of-the-mill reporting tool, the opportunities are virtually limitless! So hop on that blazing saddle as we take you to a new frontier in A/B split testing.  


Testing Overview
The design of the test is ultimately up to you, and we assume you have some experience with A/B split testing along with the math behind it.  As noted in the beginning of this article, the test design works when a single media buy key is assigned as the control per test group, and 1 or more media buys are designated as test media buys.  In this particular article, conversion rate will be the success metric which is being tested. The solution discussed is for standard split testing (not multivariate testing). Our test rests on the premise of calculating the Z score, and then using a Z score table to arrive at the correct p-value.  


The Maths & Stats behind it all
The formula used for calculating Z score:


Standard Error Formula:


Identifying Test Groups and Test IDs
In order to conduct a proper split test, we’ll need to identify which placements (media buys) belong to the test group, and which belong to the control group.  We also assume that you’ll want to run multiple A/B split tests, so for placements that are a part of a test, we’ll need to assign a test ID. Here’s the logic:

  1. Determine if placement is a part of a test (if yes proceed to step 2).

  2. Determine if the placement be placed into the test or control group.  

  3. Assign a unique test ID to all placements belong to a particular test.  

Here’s how this logic would play out with a sample data set.  Note that every media buy is not required to be assigned to a test (test group and test id fields remain null).


Next, you’ll need a way to categorize your various media buys by test group and assign the test IDs.  I’ve found the easiest way to accomplish this is with the newly release data classification (under harmonization center and currently in public beta), where a lookup table contains the media buy keys for any placement belonging to a test, along with their respective test group and test ID assignments (see screenshot below).  

Sample lookup table:



(Scroll down to the next reply to see part 2 of this article)

Comments

  • _ben_easley_ben_easley SFOSYS_ADMIN IMG42
    edited May 2019

    Part 2 of 2


    Step-by-Step Implementation Instructions
    *All formulas and their settings can be downloaded HERE.  Be sure to pay particularly close attention to the advance settings pertaining to granularity and aggregation.


    1.
     In order to appropriately calculate our conversion rate for our test and control media buys, we’ll need to create the following calculated metrics (reference the doc above): 

    - Impressions (control)
    - Impressions (test)
    - Conversions (control)
    - Conversions (test)
    - Conversion Rate (control)
    - Conversion Rate (test)
    - Conv Rate Delta (test - control)

    2.  Once you’ve created the above metrics,  you’ll want to check your work in order to ensure that all of your calculated metrics are working as expected.  For a media buy belonging to the test group, the fields Impressions (control) and Conversions (control) should return 0s.  Part of the reason that we set up each individual field (as opposed to building all of the logic into a field like Conversion Rate (Control), is so that we can QA our work.  

    Check the validity of each field using the table widget in order to see if you get the expected result:  



    3.
      One of the building blocks for calculating the Z score is obtaining the SE (standard error).  Create the below two calculated fields according to the formula document.

    - SE (control)
    - SE (test)

    4.  Next, we need to check to ensure that our SE fields are working as expected.  We assume that you checked all of the new calculated fields in step 2, as you’ll need to reference some of those fields in your calculation.  

    Run through the SE formulas by hand to confirm they match with that platform.

    5.  Now that you’ve created the calculated metrics for Conversion Rate Delta and SE (test and control), you’re ready to create the Z score calculated metric field.  Reference the formula doc, and then check your work to ensure the Z score is returning the correct value.

    6.  We’ve got our Z score.  The only missing piece is the p value.  Reference the formula document once more.  Depending on how your agency/business approaches statistical significance, you may adjust this formula to your organization's needs.  Our formula assumes that the end user is interested in the following p values:

    p >=  0.1
    p >=  0.05
    p =  0 (returns when p is between -.1 and .1)
    p <=  -0.1
    p <=  -.05

    Bonus (Calculating Lift)
    Want to calculate the lift for your test?  No problem! Just reference lift in the formula document.  You may also consider adding conditional logic to only return the lift when a certain level of statistical significance is met (the current formula returns lift regardless of the p-value)

    Programmatic Testing & Optimization
    Now that you’ve setup your workspace for A/B testing, why not take advantage of Activation Center to turn off any placement that fails to outperform the control?  If you’re running campaigns in Google Ads, Facebook, or Twitter, you can create action rules (based on the Z score or P-value fields) that trigger an action when a specific condition is met.  For example, if P-value is <= 0 for the last 90 days, trigger an action to turn off the respective Facebook ad set(s). We’d also encourage you to implement filters that consider the appropriate time frame and minimum impression delivery.   


Sign In or Register to comment.