💎 Testing velocity, success rate and average impact

The throughput funnel for a team is a function of velocity, success rate and average impact:

• Velocity: how many feature experiences are you putting out per engineer per week? Example at Airbnb: 1 test/engineer per week.
• Success rate: for 100 experiments, how many are a statistically significant winner that you’re going to roll out? A good benchmark is a 20-30% success rate which means you should expect 70-80% of your test to be flat or negative.
• Average Impact: if you have a 30% success rate, what’s the average impact over these?