Justifying Prior Choice - Deepstash

Justifying Prior Choice

Critics of a Bayesian analysis might argue that the choice of a prior distribution was not sufficiently justified and had a significant impact on the experiment. In fact, the simulation presented in the previous section assumed that we used the perfect prior distribution.

2 STASHED

MORE IDEAS FROM THEARTICLE

Bayesian Methods

In Bayesian A/B testing, we model the metric for each variant as a random variable with some probability distribution.

By accepting variants that offer a small improvement, Bayesian A/B testing asserts that the false positive rate — the proportion of times we accept the treatment when the treatment is not actually better — is not very important. 

2 STASHED

Using relevant prior information makes experiments conclude faster. Every piece of information that we embed into the prior is a piece of information that we do not need to learn from the data. By leveraging priors, Bayesian A/B testing often needs fewer data points to reach a conclusion than other methods.

2 STASHED

In any A/B test, we use the data we collect from variants A and B to compute some metric for each variant (e.g. the rate at which a button is clicked). Then, we use a statistical method to determine which variant is better.

2 STASHED

Deepstash helps you become inspired, wiser and productive, through bite-sized ideas from the best articles, books and videos out there.

GET THE APP:

RELATED IDEAS

At Netflix, running A/B tests, where possible, allows us to substantiate causality and confidently make changes to the product knowing that our members have voted for them with their actions.

An A/B test starts with an idea — some change we can make to the UI, the personalization systems that help members find content, the signup flow for new members, or any other part of the Netflix experience that we believe will produce a positive result for our members.

2 STASHED

Statistics is using math to do technical analysis of data. Instead of guesstimating, data helps us get concrete and factual information.

The most widely used statistical concept in data science is called Statistical Features. It includes important measurements like bias, variance, mean, median and percentiles. It’s all code-friendly too.

64 STASHED

10 LIKES

A/B Testing

Over the years, conversion rate optimization (CRO) seems to have become synonymous with A/B testing in the minds of many marketers.

A/B testing is a form of conversion rate optimization. You have a page and you want it to perform better, so you change something and see if it improves your results.

But, A/B testing isn’t the only way to do CRO.

If you’ve got enough traffic, multivariate testing can allow you to produce meaningful results much more quickly.

1 STASHED

❤️ Brainstash Inc.