In any A/B test, we use the data we collect from variants A and B to compute some metric for each variant (e.g. the rate at which a button is clicked). Then, we use a statistical method to determine which variant is better.
In Bayesian A/B testing, we model the metric for each variant as a random variable with some probability distribution.
By accepting variants that offer a small improvement, Bayesian A/B testing asserts that the false positive rate — the proportion of times we accept the treatment when the treatment is not actually better — is not very important.
Critics of a Bayesian analysis might argue that the choice of a prior distribution was not sufficiently justified and had a significant impact on the experiment. In fact, the simulation presented in the previous section assumed that we used the perfect prior distribution.
Using relevant prior information makes experiments conclude faster. Every piece of information that we embed into the prior is a piece of information that we do not need to learn from the data. By leveraging priors, Bayesian A/B testing often needs fewer data points to reach a conclusion than other methods.
MORE LIKE THIS
❤️ Brainstash Inc.