<ul><li class="ql-align-justify">When implementing a UBP model, prioritize allowing for maximum usage flexibility and pricing transparency for your customer. </li><li class="ql-align-justify">Create the option for customers to commit to your product for an extended time period for a pricing discount.  </li><li class="ql-align-justify">Emphasize the benefits of UBP to your investors and get them excited about usage-based metrics. </li><li class="ql-align-justify">Don’t hesitate to automate financial processes when you are ready to scale.</li><li class="ql-align-justify">Don’t let your financial team exist in a vacuum. Collaborating with outside teams will help your team do their job more accurately and be prepared for change.</li></ul>

Usage-Based Pricing(UBP)

Usage-based pricing removed a lot of friction for adoption.For customers, getting started is easy and with an emphasis on flexible usage, staying with the company becomes a no-brainer.Pricing transparency is also a huge selling point.

Removing Friction By Being Transparent

Answering questions about what the top line is going to grow to as new logos are added or as existing logos expand can be more challenging with UBP.

Manage Expectations for Predictability

<ul><li>Your product and platform's intrinsic virality gives your investors confidence in continual top-line growth.</li><li>ARR is the metric that communicates how Datadog is performing today.</li><li>Usage-based annual recurring revenue (UARR) measures usage growth over time with any vertical customer.</li></ul>

Sell Your Investors on UBP With Redefined Metrics

Financial processes need to clearly present different commitment and drawdown plans and track customer-specific contract terms.Whereas seat-based models can implement a third-party customer billing platform with relative ease, these billing platforms cannot readily account for all the features.

Automate Financial Processes to Scale

<ul><li>Pricing will change with a UBP model.</li><li>The finance team has to be able to adapt and move as these new insights arrive.</li><li>Take proactive measures to understand potential changes and prepare your teams for them.</li></ul>

What The Finance Team Needs To Do

I joined Datadog as VP Finance back in 2015 when the company was still very small. Back then, the company had about 100 employees, was making around $20

Stashed ideas

Implementing Usage-Based Pricing: What Your Financial Teams Need to Know - OpenView

<ul><li>Two of the fastest-growing categories among the social apps were friend discovery and vertical communities</li><li>Will the future of social media be interest-based platforms that foster meaningful connection?</li></ul>

Results

<ul><li>Usually, Daily Active Users (DAU) and Monthly Active Users (MAU) are used to measure scale and growth</li><li>DAU/MAU=stickiness. Can be used to evaluate an app’s network effects, or how strong the network is based on how frequently its users engage.</li><li>DAU/MAU is helpful but isn’t all-encompassing—often will be higher around media consumption</li><li>Addiction (number of times app is accessed a day) and absorption (how long each session on the app lasts) are also important metrics that were analyzed. The key is the combination of the two given by the attention equation: frequency+depth=total time spent</li></ul>

Social App Metrics

<ul><li>Every app’s success benchmarks are specific to the app’s category</li><li>Dating apps should be compared to other dating apps, not TikTok or Facebook, etc.</li></ul>

Benchmarks by Category

<ul><li>Fastest growing social apps</li><li>Friend discovery—teens turn to social apps to make and build friendships</li><li>Vertical communities—the value of going niche; social networks formed around specific interests</li><li>Combination of utility and community results in high retention and downstream monetization opportunities</li></ul>The best social apps are those with an intrinsic social component. Measuring success can be difficult and there is more than one path. Category matters for benchmarks!

Friend discovery apps and vertical communities

Evaluating the potential of a social app can be complicated. Many factors must be considered. Using Apptopia, social platforms in the top 1,000 on the US App store were analyzed and ranked according to growth rate (see graphics in-article).

When assessing a social app’s potential, what does “good” look like, anyway? The most important metrics—and benchmarks to aspire to—according to the data.

The Stickiest, Most Addictive, Most Engaging, and Fastest-Growing Social Apps—and How to Measure Them | Andreessen Horowitz

Many C-suite executives, supervisors, and managers are fixated on metrics to evaluate overall company performance and effectiveness. Employees are reminded that "What gets measured gets done." Yet, there is another truth that is equally true. Jerry Muller, author of The Tyranny of Metrics: “Not everything that is important is measurable, and much that is measurable is unimportant.”  While metrics can help us stay focused on goals, they can also be dangerous.

Leaning on metrics can be dangerous

It's tempting to only rely on metrics that are easy to measure, but they often give us only half the story. An obsession with performance indicators can make organizations rigid, shortsighted, and dismissive of innovation. Data around reputation, loyalty, trust, satisfaction and more are also not easy to capture and may be misleading. For example, consider an employee who performs well in front of a demanding client but gets a lower feedback rating.

Metrics can't give us a complete picture

The best way to build a culture where your team buys into the goals and feel intrinsically motivated starts with trust. Most companies measure trust by tacit approval and execution of plans or a lack of discussion on the goals. But that is not trust; it's obedience. Trust is about safety. A better measure of safety is a willingness to speak the truth and be prepared to disagree with an idea, and have a willingness to let ideas win over authority. There must be room for failure, experimentation, innovation, and questions.

Build a sense of trust

Metrics can create a false sense of security. For example, Peloton held on to a near-rabid fandom as evidence of their success. When regulators started to question the safety of the company's treadmills, they focused harder on the strength of their brand and attacked the agency and the accuracy of the reports. By February, the stock fell almost 70%, the CEO was replaced by the board, and nearly 2,800 employees were being laid off. However, the marketing and manufacturing employees knew of the company issues but were not empowered to make the right decisions.

Prioritise feedback and flexibility

How to lead without focus on metrics

You need to select the right metrics to measure the success of the product or feature you're building.Once selected, metrics help the product development in many ways:<ul><li>They provide direction for the product development</li><li>They offer data to make better decisions</li><li>They help measure the impact of new features</li><li>They provide alignment across teams</li><li>They help identify problems</li></ul>

Benefits of product metrics

Answer these questions to find the right metrics for your product:<ul><li>Primary output metric — When will you call your product a success or failure?</li><li>Input metrics — What milestones will your product hit before it becomes a success?</li><li>Check metrics — Are you screwing up other company metrics while building your product?</li><li>External dependent metrics — What external metrics outside of your control are required for your product to be a success?</li></ul>Tap the image to see example metrics for improving revenue/ride at Uber.There are two more examples in the article if you're curious (Netflix and Amazon Prime).

The right set of metrics

<ul><li>Define metrics in such a way that they are measurable and can be tracked on a regular basis</li><li>Define the increment in the metric that your product will bring: "Increase average session length from 5min to 8min"</li><li>Define the time period in which that metric will be achieved: "Increase survey completion rate from 2% to 5% within 3 months"</li></ul>

Best practices for defining metrics

The difference between a successful product and an unsuccessful product is the selection of the right metric to measure the success of the product. Without a measurable success metric your product is…

Product Metrics

Input-Output Metrics for Product Managers

Not only do we typically work with very large data sets, but those data sets are extremely rich. That is, each row of data typically has many, many attributes. When you combine this with the temporal sequences of events for a given user, there are an enormous number of ways of looking at the data. 
Contrast this with a typical academic psychology experiment where it's trivial for the researcher to look at every single data point. The problems posed by our large, high-dimensional data sets are very different from those encountered throughout most of the history of scientific work.

New Frontiers In Data Analysis

Most practitioners use summary metrics (for example, mean, median, standard deviation, and so on) to communicate about distributions. 
However, you should usually examine much richer distribution representations by generating histograms, cumulative distribution functions (CDFs), Quantile-Quantile (Q-Q) plots, and so on. These richer representations allow you to detect important features of the data, such as multimodal behavior or a significant class of outliers.

Data Distributions

Examine outliers carefully because they can be canaries in the coal mine that indicate more fundamental problems with your analysis. 
It's fine to exclude outliers from your data or to lump them together into an "unusual" category, but you should make sure that you know why data ended up in that category.

Consider The Outliers

Randomness exists and will fool us. Some people think, “Google has so much data; the noise goes away.” This simply isn’t true. Every number or summary of data that you produce should have an accompanying notion of your confidence in this estimate (through measures such as confidence intervals and p-values).

Look At The Noise

Anytime you are producing new analysis code, you need to look at examples from the underlying data and how your code is interpreting those examples. Your analysis is abstracting away many details from the underlying data to produce useful summaries.
How you sample these examples is important:
<ul>
<li>If you are classifying the underlying data, look at examples belonging to each class.</li>
<li>If it's a bigger class, look at more samples.</li>
<li>If you are computing a number (for example, page load time), make sure that you look at extreme examples</li>
</ul>

Look At Examples

Slicing means separating your data into subgroups and looking at metric values for each subgroup separately. We commonly slice along dimensions like browser, locale, domain, device type, and so on. If the underlying phenomenon is likely to work differently across subgroups, you must slice the data to confirm whether that is indeed the case. 
Even if you do not expect slicing to produce different results, looking at a few slices for internal consistency gives you greater confidence that you are measuring the right thing.

Slice Your Data

With a large volume of data, it can be tempting to focus solely on statistical significance or to home in on the details of every bit of data. But you need to ask yourself, "Even if it is true that value X is 0.1% more than value Y, does it matter?" 
This can be especially important if you are unable to understand/categorize part of your data. If you are unable to make sense of some user-agent strings in your logs, whether it represents 0.1% or 10% of the data makes a big difference in how much you should investigate those cases.

Consider Practical Significance

You should almost always try slicing data by units of time because many disturbances to underlying data happen as our systems evolve over time. (We often use days, but other units of time may also be useful.)
During the initial launch of a feature or new data collection, practitioners often carefully check that everything is working as expected. However, many breakages or unexpected behavior can arise over time.

Check For Consistency Over Time

Looking at day-over-day data also gives you a sense of the variation in the data that would eventually lead to confidence intervals or claims of statistical significance. This should not generally replace rigorous confidence-interval calculation, but often with large changes you can see they will be statistically significant just from the day-over-day graphs.

Check For Variation

Almost every large data analysis starts by filtering data in various stages. Maybe you want to consider only US users, or web searches, or searches with ads. Whatever the case, you must:
<ul>
<li>Acknowledge and clearly specify what filtering you are doing.</li>
<li>Count the amount of data being filtered at each step.</li>
<li>Often the best way to do the latter is to compute all your metrics, even for the population you are excluding. You can then look at that data to answer questions like, "What fraction of queries did spam filtering remove?"</li>
</ul>

Acknowledge And Count Your Filtering

The most interesting metrics are ratios of underlying measures. Oftentimes, interesting filtering or other data choices are hidden in the precise definitions of the numerator and denominator. For example, which of the following does “Queries / User” actually mean?
<ul>
<li>Queries / Users with a Query</li>
<li>Queries / Users who visited Google today</li>
<li>Queries / Users with an active account (yes, I would have to define active)</li>
<li>Being really clear here can avoid confusion for yourself and others.</li>
</ul>

Ratios Should Have Clear Numerators And Denominators

Validation: Do I believe the data is self-consistent, that it was collected correctly, and that it represents what I think it does?Description: What's the objective interpretation of this data? For example, "Users make fewer queries classified as X," "In the experiment group, the time between X and Y is 1% larger," and "Fewer users go to the next page of results."Evaluation: Given the description, does the data tell us that something good is happening for the user, for Google, or for the world?

Three Stages Of Data Analysis: Validation, Description, and Evaluation

Before looking at any data, make sure you understand the context in which the data was collected. If the data comes from an experiment, look at the configuration of the experiment. If it's from new client instrumentation, make sure you have at least a rough understanding of how the data is collected. 
You may spot unusual/bad configurations or population restrictions (such as valid data only for Chrome). Anything notable here may help you build and verify theories later.

Confirm Experiment And Data Collection Setup

As part of the "Validation" stage, before actually answering the question you are interested in (for example, "Did adding a picture of a face increase or decrease clicks?"), rule out any other variability in the data that might affect the experiment. For example:
<ul>
<li>Did the number of users change?</li>
<li>Did the right number of affected queries show up in all my subgroups?</li>
<li>Did error rates change?</li>
</ul>
These questions are sensible both for experiment/control comparisons and when examining trends over time.

Check For What Shouldn't Change

When looking at new features and new data, it's particularly tempting to jump right into the metrics that are new or special for this new feature. However, you should always look at standard metrics first, even if you expect them to change. 
For example, when adding a new universal block to the page, make sure you understand the impact on standard metrics like “clicks on web results” before diving into the custom metrics about this new result.

Standard First, Custom Second

Especially if you are trying to capture a new phenomenon, try to measure the same underlying thing in multiple ways. Then, determine whether these multiple measurements are consistent. 
By using multiple measurements, you can identify bugs in measurement or logging code, unexpected features of the underlying data, or filtering steps that are important. It’s even better if you can use different data sources for the measurements.

Measure Twice, Or More

Both slicing and consistency over time are particular examples of checking for reproducibility. If a phenomenon is important and meaningful, you should see it across different user populations and time. But verifying reproducibility means more than performing these two checks. If you are building models of the data, you want those models to be stable across small perturbations in the underlying data.
If a model is not reproducible, you are probably not capturing something fundamental about the underlying process that produced the data.

Check For Reproducibility

Often you will be calculating a metric that is similar to things that have been counted in the past. You should compare your metrics to metrics reported in the past, even if these measurements are on different user populations.
You do not need to get an exact agreement, but you should be in the same ballpark. If you are not, assume that you are wrong until you can fully convince yourself. Most surprising data will turn out to be an error, not a fabulous new insight.

Check For Consistency With Past Measurements

If you create new metrics (possibly by gathering a novel data source) and try to learn something new, you won’t know if your new metric is right. With new metrics, you should first apply them to a known feature or data.
If you have a new metric for where users are directing their attention to the page, make sure it matches what we know from looking at eye-tracking or rater studies about how images affect page attention. Doing this provides validation when you then go to learn something new.

New Metrics Should Be Applied To Old Data/Features First

Typically, data analysis for a complex problem is iterative. You will discover anomalies, trends, or other features of the data. Naturally, you will develop theories to explain this data. Don’t just develop a theory and proclaim it to be true. Look for evidence (inside or outside the data) to confirm/deny this theory.

Looking For Evidence

Good data analysis will have a story to tell. To make sure it’s the right story, you need to tell the story to yourself, then look for evidence that it’s wrong. One way of doing this is to ask yourself, “What experiments would I run that would validate/invalidate the story I am telling?” Even if you don’t/can’t do these experiments, it may give you ideas on how to validate with the data that you do have.

A Story To Tell

When doing exploratory analysis, perform as many iterations of the whole analysis as possible. Typically you will have multiple steps of signal gathering, processing, modeling, etc. If you spend too long getting the very first stage of your initial signals perfect, you are missing out on opportunities to do more iterations in the same amount of time. 
Further, when you finally look at your data at the end, you may make discoveries that change your direction. Therefore, your initial focus should not be on perfection but on getting something reasonable all the way through.

Exploratory Analysis Benefits From End-To-End Iteration

We typically define various metrics around user success.
You can not use the metric that is fed back to your system as a basis for evaluating your change. If you show more ads that get more clicks, you can not use “more clicks” as a basis for deciding that users are happier, even though “more clicks” often means “happier.” Further, you should not even do slicing on the variables that you fed back and manipulated, as that will result in mixed shifts that will be difficult or impossible to understand.

Watch Out For Feedback

<ul><li>As you work with data, you must become both the champion of the insights you are gaining and a skeptic of them. You will hopefully find some interesting phenomena in the data you look at. When you detect an interesting phenomenon, ask yourself the following questions:</li><li>What other data could I gather to show how awesome this is?</li><li>What could I find that would invalidate this?</li></ul>

Skeptic And Champion At One Go

There’s always a motivation to analyze data. Formulating your needs as questions or hypotheses helps ensure that you are gathering the data you should be gathering and that you are thinking about the possible gaps in the data. Of course, the questions you ask should evolve as you look at the data. However, analysis without a question will end up aimless.
Avoid the trap of finding some favourite technique and then only finding the parts of problems that this technique works on. Again, creating clear questions will help you avoid this trap.

Data Analysis Starts With Questions, Not Data Or A Technique

When making theories about data, we often want to assert that "X causes Y"—for example, "the page getting slower caused users to click less." You can not simply establish causation because of correlation. By considering how you would validate a theory of causation, you can usually develop a good sense of how credible a causal theory is.Sometimes, people try to hold on to a correlation as meaningful by asserting that even if there is no causal relationship between A and B, there must be something underlying the coincidence so that one signal can be a good indicator or proxy for the other

Correlation and Causation

The previous points suggested some ways to get yourself to do the right kinds of soundness checking and validation. But sharing with a peer is one of the best ways to force yourself to do all these things. A skilled peer can provide qualitatively different feedback than the consumers of your data can. Peers are useful at multiple points through the analysis. 
Early on you can find out about gotchas your peer knows about, suggestions for things to measure, and past research in this area. Near the end, peers are very good at pointing out oddities, inconsistencies, or other confusions.

Bite-sized knowledge
to upgrade
your career

#metrics

IDEAS ABOUT #metrics

Implementing Usage-Based Pricing: What Your Financial Teams Need to Know - OpenView

The Stickiest, Most Addictive, Most Engaging, and Fastest-Growing Social Apps—and How to Measure Them | Andreessen Horowitz

It's time to

READ

LIKE

A PRO!

How to lead without focus on metrics

Input-Output Metrics for Product Managers

Good Data Analysis | ML Universal Guides | Google Developers

Bite-sized knowledgeto upgradeyour career

#metrics

IDEAS ABOUT #metrics

Implementing Usage-Based Pricing: What Your Financial Teams Need to Know - OpenView

The Stickiest, Most Addictive, Most Engaging, and Fastest-Growing Social Apps—and How to Measure Them | Andreessen Horowitz

It's time to

READ

LIKE

A PRO!

How to lead without focus on metrics

Input-Output Metrics for Product Managers

Good Data Analysis | ML Universal Guides | Google Developers

Bite-sized knowledge
to upgrade
your career