quantuity

quantuity / BayesianSplitTest / 1.1.0

README.md

Overview

Bayesian Split Testing is an alternative method to T-Testing, to analyse the output of a split test. Rather than measuring significance, Bayesian Split Testing returns the measured lift and the probability that the test treatment is better than the control treatment.

Applicable Scenarios and Problems

In a split testing the audience is split into 2 groups, with each group being given a different treatment. Usually one treatment is the existing treatment (control) and the other treatment is an alternative treatment (test). In conventional T-Testing, the test duration needs to be decided prior to running the experiment and the test measures if the test and control treatments resulted in outcomes to a given significance level. T-Tests require large samples and can be inconclusive. T-Tests should not be run on while an experiment is in progress and must wait until the experiment duration has ended. This can lead to an opportunity cost on occasions where the test variant far exceeds expectations.

Bayesian Split Testing is an alternative approach which can be applied at any point during an experiment without interfering with the experiment results. Given the observations up to that point Bayesian Split Testing will return the observed lift and the probability that the treatment received by the test variant is better than that the treatment received by the control variant, or in short the probability than the test is better than the control. Used from the beginning of the experiment and used at several points, or continuously during the experiment duration, Bayesian Split Testing offers a method to make a fair, impartial decisions as to the success of a test treatment at an early stage on an experiment and avoid the opportunity cost of running the experiment for longer than necessary.

Usage

Input

Input taken in the form of a list:

[Count of Control Variant Successes, Count of Control Variant Samples, Count of Test Variant Success, Count of Test Variant Samples]

Output

Returns the lift of the Test Variant over the Control Variant and the probability that the Test Variant is better than the Control Variant.

Examples

Given the following observations from an experiment:

VariantSuccessesObservations
Control3211000
Test3421000

Input:

[321,1000,342,1000]

Output:

[
  0.06521739130434782,
  0.8405864345137686
]

The output indicates that the observed lift of the test variant over the control variant is 6.52% and the probability that the test variant is better than the control variant is 84.06%.