Product Management — All about Metrics

10 min readMar 2, 2022

As a PM you will always need metrics prior to product development, any new A/B experience. you will be always trying to optimize towards a specific metric. The goal is to build a data-driven product.

It is always very important to pick the right metrics for each problem. If we take any of the FAANG or MANGA “Facebook or what now called Meta, Apple, Amazon, Netflix, and Google” companies and look into the different kinds of metrics they use. facebook's vision starts with “Bringing People Closer Together” which translates into people spending more time on Facebook this kind of metric we call A North Star, the metric that all company products implicitly or explicitly work towards. furthermore, a company like Facebook would use.

Engagement Metrics including:

Number of likes given/ received
Number of comments given/ received
Number of post shares given/ received
Number of created posts
Number of posts/comments/likes per user per FB group
Time spent on the site
Sent/ received private and group messages
Position likes (The position of post that received user interaction “like, comment, etc.”). This is considered an important internal metric PM could use to make sure that the team is on the right way and really achieving relevancy, in other words, users can relatively quick encounter content that interest them and is worthy of their interaction.

It is important to measure received messages, likes, shares, and comments along with the given ones as you would want to make sure to measure the quantized amount of interaction users get so you can assure they having a good time on FB which as a result will make them spend more time.

Growth Metrics including:

The total number of users vs active users. Active users are identified in different ways for different products, some consider them people who sign up in order to use your product and actually use it at least once a week
Number of created posts
Number of created groups

I could see the Number of created posts used as both engagement and growth metrics at different points of the product life, in other words, I see FB using it as a growth metric in the early life of the product but then to measure active user engagement later.

A company like Twitter will use metrics as follows:

Engagement Metrics including:

Avg. Number of Multiple logins per day
Time spent on the site
Total Number of tweets
Avg. Number of likes, retweets, replies and follows
Sent/ received private messages

It is important for a product manager in any of the social platforms to measure against time well-spent on the site so you can make sure that users will be engaged and will always come back

Growth Metrics including:

Total number of new users/ week or month
Monthly/ Daily Active users (MAU and DAU)

If we try to categorize the metrics in buckets, we will find that it mostly is divided into:

Growth and Activation: this includes total users, total active users. You would often want to know the source of new users such information will give you more leverage to exploit your top source of new users
Engagement: this measures users' engagement and interaction with the product and it differs according to the product itself. It is often measured in ratios i.e. Active users/All users
Retention: this measures if your signed-up users are coming back and after how much time and how much time they spend when they come back. measuring retention will help you sustain your product by maintaining user interaction. Most of the time you would calculate metrics like retuning users and resurrected users (people who come back after abandoning your product and you would want to know why they came back and after how much time)
User Happiness: this measures user satisfaction with your product which, going to give you an indicator if they are so happy with your product that they will tell their friends to use it. To measure happiness companies will use things like NPS (Net-Promoter Score) which is calculated by in-app or in-email product surveys answering questions like “on a scale of 1–10 how strongly would you recommend product X for a friend?”

illustration credit — CheckMarket by Medailia

5. Revenue: this is the livelihood of every product answering the very important question. Is your product making money? to specify this you would want to measure the following:

LTV (Life Time Value): money/ revenue generated by each user through their lifetime as a paying customer
CIC/CAA (Cost of Customer Acquisition): you will target to keep this as low as possible. you will calculate this against the number of gained new customers
ARR (Anual Recurring Revenue): the revenue generated by your subscribed users on an annual basis

Illustration credit — A guide to SaaS metrics for product managers from roadmunk

Now, I would like to talk about What makes a good metric?

Strategic: this comes back to the point that your metric must always be connected to a specific product goal/vision or what we call north star that is aligned with the organization strategy
Understandable/Actionable/Relevant/Referenceable: this means that your metric should be simple and communicable for all stakeholders
Rate or Ratio: most of the time you would want to calculate metrics like active users relative to a total number of users. So, it is a good idea to have an idea of the big picture and look into numbers relative to the big picture while identifying dependencies. An example, Airbnb would measure Avg # of nights booked/person/month rather than measuring the total # of booked nights.
Correlation vs. Causation: be aware of assuming causation between 2 data points because you can see their correlation

Illustration credit — Using Lean Analytics Principles to Build a Strong Company

5. Changeable: your metric should be changeable, in other words, you can change or even control how your users use your product optimizing for this specific metric. For example, if you know that a specific customer group has only 2 hrs. only they can use your product, it is a much more simple problem to try to increase the value generated through these 2 hrs. rather than trying to increase them.

Also, be aware of the metric you choose given the nature of your product, for example, a product that is designed to provide a Doordash like service but for fine dining and 4-star restaurants cannot optimize on increasing the ordered items as a regular user is not expected to need this kind of service on a very regular basis but rather use it on special occasions, holidays, valentines, etc. So, it makes more sense to try to increase the value obtained with those irregular orders rather than trying to increase the number of orders.

What is a metric rollup?

This happens when a PM is using a metric that is contributing towards the north star goals in an indirect way, example for it if you are A PM in FB and you decide to optimize towards a metric that makes users comment more as it is already proven from the data that users who tend to comment more, spend more time on FB.

What is a counter metric? and Why do we need it?

A counter metric ensures that any improvement didn’t come at the expense of an equally important outcome or in other words, you haven’t over-optimized your north star metric to the detriment of your customers and your business
If you only measure your north star, you won’t be able to capture any negative downstream effects that could come with changes to your north star metric. I will borrow this great example from PRODUCT MANAGER HQ showing how things could go wrong optimizing only towards a better value for your north star

PMs & Metrics: Counter Metrics

While north star metrics are incredibly valuable for creating focus and alignment, they provide even more value when…

productmanagerhq.com

For example, let’s say that you’re the product manager for a website that sells Spanish language courses.
You’ve previously decided your north star metric is “number of users who visit your site every week,” because your business hypothesis is that if more users visit the site, you’ll be able to convert more of them into customers. You currently have 10,000 weekly visitors, and you convert at a 1% rate, so that means you’re selling 10,000 * 1% = 100 Spanish courses every week.
You learn that running a giveaway for prizes is a fantastic way to bring new users to any website, so you decide to give away an Oculus Rift. To get even more attention, you make the Oculus Rift giveaway the primary call to action (CTA) of your landing page. You also run social media campaigns on Snapchat.
Your north star metric shoots upwards by 30%, so now you have 13,000 visitors every week! That means you’re doing well, right?
But then, you find out that you’re now only selling 50 Spanish courses per week. How is that possible?
First, your giveaway visitors may not actually be about the core product that you’re selling on your site. So, all they do is register for the giveaway, then they leave. They never convert.
Second, your typical visitor base may have been unable to access the content that they would typically access because the giveaway takes up so much space and attention.
Your typical visitor base is looking to learn Spanish. They’re likely not interested in winning an Oculus Rift, since an Oculus Rift has nothing to do with their core pain of learning Spanish. They likely became confused and left your site because they couldn’t find what they were looking for.
In other words, not only did you fail to convert the giveaway visitors to your site, you also drove away the original audience that would have converted.

In this example, a good counter metric would be conversion rate so you would have to always measure the number of converted new users against new signed up users. A good counter metric must have all the characteristics I already mentioned picking a good metric. Check this cool table showing mostly “best” counter metric for your north star metric

Illustration credit — PRODUCT MANAGER HQ: PMs & Metrics: Counter Metrics

Exploratory vs Reporting Metrics?

As a PM you can categorize the metrics you decide to use measuring the performance of your product into exploratory and reporting metrics, you will find that most of the time exploratory metrics tend to be more on the qualitative side while the reporting metrics tend to be on the quantitative side and this is due to the nature of the target audience because the reporting metrics — as already obvious from the name, the PM report those values in regular meetings with interested stakeholders while the exploratory metrics are intended more for internal usage meaning the PM will use them to identify their next feature or backlog item to prioritize and to find unknown insight

Methods to select your perfect metric?

HEART (or more like ATERH)

This is a methodology introduced by Kerry Rodden while she was working with Google product team defining UX metrics. this method helps PM identify goals, signals, and Metrics across 5 main aspects Happiness, Engagement, Adoption, Retention, and Task Successmaking the acronym HEART. This helps PM identify success metrics in a holistic view that puts all stakeholders' concerns at the center of it.

Happiness: measures of user attitudes, often collected via survey. For example satisfaction, perceived ease of use, and net-promoter score.
Engagement: how users are using your product, in terms of frequency or number/ types of features used
Adoption: how many new users start using your product or feature. For example, the number of accounts created in the last seven days
Retention: the rate at which existing users keep coming back. For example: how many of the active users from a given time period are still present in some later time period? You may be more interested in failure to retain, commonly known as “churn.”
Task success: the ability for users to complete critical tasks, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. This category is most applicable to areas of your product that are very task-focused, such as search or an upload flow.

Across these dimensions, you will then want to start by identifying goals, then signals, and finally metrics to measure how well your product is doing.

A goal is something the user is trying to do or something you are trying to help them to do
A signal is a change in user behavior that indicates that the user is achieving the goal
A metric is a way to measure signal and quantify how much user behavior has changed

Keep in mind that it’s ok for goals to have multiple signals and metrics. It’s also not necessary to have metrics for every dimension. It’s more important to pick things that are relevant to the problem you are trying to solve.

I will try to walk you through an example for an imaginary product I am launching called “Amazon IndieLand” which is considering Amazon trial to enter the music and video self-publishing industry same as they did with Kindle books, the Product vision/ North star “Help rising artists live-off their art”

Sources