In any A/B test, we use the data we collect from variants A and B to compute some metric for each variant (e.g. the rate at which a button is clicked). Then, we use a statistical method to determine which variant is better.

2 STASHED

The Power of Bayesian A/B Testing

medium.com

Bayesian Methods

In Bayesian A/B testing, we model the metric for each variant as a random variable with some probability distribution.

By accepting variants that offer a small improvement, Bayesian A/B testing asserts that the false positive rate — the proportion of times we accept the treatment when the treatment is not actually better — is not very important. 

2 STASHED

Critics of a Bayesian analysis might argue that the choice of a prior distribution was not sufficiently justified and had a significant impact on the experiment. In fact, the simulation presented in the previous section assumed that we used the perfect prior distribution.

2 STASHED

Using relevant prior information makes experiments conclude faster. Every piece of information that we embed into the prior is a piece of information that we do not need to learn from the data. By leveraging priors, Bayesian A/B testing often needs fewer data points to reach a conclusion than other methods.

2 STASHED

Deepstash helps you become inspired, wiser and productive, through bite-sized ideas from the best articles, books and videos out there.

GET THE APP: