Keep reading for FREE
A/B tests let us make causal statements. We’ve introduced the Upside Down product experience to Group B only, and because we’ve randomly assigned members to groups A and B, everything else is held constant between the two groups. We can therefore conclude with high probability (more on the details next time) that the Upside Down product caused the reduction in engagement.
5
9 reads
MORE IDEAS ON THIS
To run the experiment, we take a subset of our members, usually a simple random sample, and then use random assignment to evenly split that sample into two groups. Group “A,” often called the “control group,” continues to receive the base Netflix UI experience, while Group “B,” often called the “...
5
16 reads
With many experiments, including the Upside Down box art example, we need to think carefully about what our metrics are telling us. Suppose we look at the click-through rate, measuring the fraction of members in each experience that clicked on a title. This metric alone may be a misleading measur...
5
13 reads
Because we create our control (“A”) and treatment (“B”) groups using random assignment, we can ensure that individuals in the two groups are, on average, balanced on all dimensions that may be meaningful to the test. Random assignment ensures, for example, that the average length of Netflix membe...
5
16 reads
At Netflix, running A/B tests, where possible, allows us to substantiate causality and confidently make changes to the product knowing that our members have voted for them with their actions.
An A/B test starts with an idea — some change we can make to the UI, the personalization systems th...
5
63 reads
With the Top 10 example, the hypothesis read: “Showing members the Top 10 experience will help them find something to watch, increasing member joy and satisfaction.” The primary decision metric for this test (and many others) is a measure of member engagement with Netflix: are the ideas we are te...
5
11 reads
CURATED FROM
Explore the World’s
Best Ideas
Save ideas for later reading, for personalized stashes, or for remembering it later.
Start
31 ideas
Start
44 ideas
# Personal Growth
Take Your Ideas
Anywhere
Just press play and we take care of the words.
No Internet access? No problem. Within the mobile app, all your ideas are available, even when offline.
Ideas for your next work project? Quotes that inspire you? Put them in the right place so you never lose them.
Start
47 ideas
Start
75 ideas
My Stashes
Join
2 Million Stashers
4.8
5,740 Reviews
App Store
4.7
72,690 Reviews
Google Play
Sean Green
Great interesting short snippets of informative articles. Highly recommended to anyone who loves information and lacks patience.
“
Ashley Anthony
This app is LOADED with RELEVANT, HELPFUL, AND EDUCATIONAL material. It is creatively intellectual, yet minimal enough to not overstimulate and create a learning block. I am exceptionally impressed with this app!
“
Shankul Varada
Best app ever! You heard it right. This app has helped me get back on my quest to get things done while equipping myself with knowledge everyday.
“
samz905
Don’t look further if you love learning new things. A refreshing concept that provides quick ideas for busy thought leaders.
“
Ghazala Begum
Even five minutes a day will improve your thinking. I've come across new ideas and learnt to improve existing ways to become more motivated, confident and happier.
“
Jamyson Haug
Great for quick bits of information and interesting ideas around whatever topics you are interested in. Visually, it looks great as well.
“
Giovanna Scalzone
Brilliant. It feels fresh and encouraging. So many interesting pieces of information that are just enough to absorb and apply. So happy I found this.
“
Laetitia Berton
I have only been using it for a few days now, but I have found answers to questions I had never consciously formulated, or to problems I face everyday at work or at home. I wish I had found this earlier, highly recommended!
“
Read & Learn
20x Faster
without
deepstash
with
deepstash
with
deepstash
Access to 200,000+ ideas
—
Access to the mobile app
—
Unlimited idea saving & library
—
—
Unlimited history
—
—
Unlimited listening to ideas
—
—
Downloading & offline access
—
—
Personalized recommendations
—
—
Supercharge your mind with one idea per day
Enter your email and spend 1 minute every day to learn something new.
I agree to receive email updates