Product Management — All about Metrics

  • Number of likes given/ received
  • Number of comments given/ received
  • Number of post shares given/ received
  • Number of created posts
  • Number of posts/comments/likes per user per FB group
  • Time spent on the site
  • Sent/ received private and group messages
  • Position likes (The position of post that received user interaction “like, comment, etc.”). This is considered an important internal metric PM could use to make sure that the team is on the right way and really achieving relevancy, in other words, users can relatively quick encounter content that interest them and is worthy of their interaction.
  • The total number of users vs active users. Active users are identified in different ways for different products, some consider them people who sign up in order to use your product and actually use it at least once a week
  • Number of created posts
  • Number of created groups
  • Avg. Number of Multiple logins per day
  • Time spent on the site
  • Total Number of tweets
  • Avg. Number of likes, retweets, replies and follows
  • Sent/ received private messages
  • Total number of new users/ week or month
  • Monthly/ Daily Active users (MAU and DAU)
  1. Growth and Activation: this includes total users, total active users. You would often want to know the source of new users such information will give you more leverage to exploit your top source of new users
  2. Engagement: this measures users' engagement and interaction with the product and it differs according to the product itself. It is often measured in ratios i.e. Active users/All users
  3. Retention: this measures if your signed-up users are coming back and after how much time and how much time they spend when they come back. measuring retention will help you sustain your product by maintaining user interaction. Most of the time you would calculate metrics like retuning users and resurrected users (people who come back after abandoning your product and you would want to know why they came back and after how much time)
  4. User Happiness: this measures user satisfaction with your product which, going to give you an indicator if they are so happy with your product that they will tell their friends to use it. To measure happiness companies will use things like NPS (Net-Promoter Score) which is calculated by in-app or in-email product surveys answering questions like “on a scale of 1–10 how strongly would you recommend product X for a friend?”
illustration credit — CheckMarket by Medailia
  • LTV (Life Time Value): money/ revenue generated by each user through their lifetime as a paying customer
  • CIC/CAA (Cost of Customer Acquisition): you will target to keep this as low as possible. you will calculate this against the number of gained new customers
  • ARR (Anual Recurring Revenue): the revenue generated by your subscribed users on an annual basis
Illustration credit — A guide to SaaS metrics for product managers from roadmunk
  1. Strategic: this comes back to the point that your metric must always be connected to a specific product goal/vision or what we call north star that is aligned with the organization strategy
  2. Understandable/Actionable/Relevant/Referenceable: this means that your metric should be simple and communicable for all stakeholders
  3. Rate or Ratio: most of the time you would want to calculate metrics like active users relative to a total number of users. So, it is a good idea to have an idea of the big picture and look into numbers relative to the big picture while identifying dependencies. An example, Airbnb would measure Avg # of nights booked/person/month rather than measuring the total # of booked nights.
  4. Correlation vs. Causation: be aware of assuming causation between 2 data points because you can see their correlation
Illustration credit — Using Lean Analytics Principles to Build a Strong Company

What is a metric rollup?

  • This happens when a PM is using a metric that is contributing towards the north star goals in an indirect way, example for it if you are A PM in FB and you decide to optimize towards a metric that makes users comment more as it is already proven from the data that users who tend to comment more, spend more time on FB.

What is a counter metric? and Why do we need it?

  • A counter metric ensures that any improvement didn’t come at the expense of an equally important outcome or in other words, you haven’t over-optimized your north star metric to the detriment of your customers and your business
  • If you only measure your north star, you won’t be able to capture any negative downstream effects that could come with changes to your north star metric. I will borrow this great example from PRODUCT MANAGER HQ showing how things could go wrong optimizing only towards a better value for your north star

For example, let’s say that you’re the product manager for a website that sells Spanish language courses.

You’ve previously decided your north star metric is “number of users who visit your site every week,” because your business hypothesis is that if more users visit the site, you’ll be able to convert more of them into customers. You currently have 10,000 weekly visitors, and you convert at a 1% rate, so that means you’re selling 10,000 * 1% = 100 Spanish courses every week.

You learn that running a giveaway for prizes is a fantastic way to bring new users to any website, so you decide to give away an Oculus Rift. To get even more attention, you make the Oculus Rift giveaway the primary call to action (CTA) of your landing page. You also run social media campaigns on Snapchat.

Your north star metric shoots upwards by 30%, so now you have 13,000 visitors every week! That means you’re doing well, right?

But then, you find out that you’re now only selling 50 Spanish courses per week. How is that possible?

First, your giveaway visitors may not actually be about the core product that you’re selling on your site. So, all they do is register for the giveaway, then they leave. They never convert.

Second, your typical visitor base may have been unable to access the content that they would typically access because the giveaway takes up so much space and attention.

Your typical visitor base is looking to learn Spanish. They’re likely not interested in winning an Oculus Rift, since an Oculus Rift has nothing to do with their core pain of learning Spanish. They likely became confused and left your site because they couldn’t find what they were looking for.

In other words, not only did you fail to convert the giveaway visitors to your site, you also drove away the original audience that would have converted.

In this example, a good counter metric would be conversion rate so you would have to always measure the number of converted new users against new signed up users. A good counter metric must have all the characteristics I already mentioned picking a good metric. Check this cool table showing mostly “best” counter metric for your north star metric

Illustration credit — PRODUCT MANAGER HQ: PMs & Metrics: Counter Metrics

Exploratory vs Reporting Metrics?

As a PM you can categorize the metrics you decide to use measuring the performance of your product into exploratory and reporting metrics, you will find that most of the time exploratory metrics tend to be more on the qualitative side while the reporting metrics tend to be on the quantitative side and this is due to the nature of the target audience because the reporting metrics — as already obvious from the name, the PM report those values in regular meetings with interested stakeholders while the exploratory metrics are intended more for internal usage meaning the PM will use them to identify their next feature or backlog item to prioritize and to find unknown insight

Methods to select your perfect metric?

HEART (or more like ATERH)

This is a methodology introduced by Kerry Rodden while she was working with Google product team defining UX metrics. this method helps PM identify goals, signals, and Metrics across 5 main aspects Happiness, Engagement, Adoption, Retention, and Task Successmaking the acronym HEART. This helps PM identify success metrics in a holistic view that puts all stakeholders' concerns at the center of it.

  • Happiness: measures of user attitudes, often collected via survey. For example satisfaction, perceived ease of use, and net-promoter score.
  • Engagement: how users are using your product, in terms of frequency or number/ types of features used
  • Adoption: how many new users start using your product or feature. For example, the number of accounts created in the last seven days
  • Retention: the rate at which existing users keep coming back. For example: how many of the active users from a given time period are still present in some later time period? You may be more interested in failure to retain, commonly known as “churn.”
  • Task success: the ability for users to complete critical tasks, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. This category is most applicable to areas of your product that are very task-focused, such as search or an upload flow.
  • A goal is something the user is trying to do or something you are trying to help them to do
  • A signal is a change in user behavior that indicates that the user is achieving the goal
  • A metric is a way to measure signal and quantify how much user behavior has changed
Amazon IndieLand HEART metrics table
  1. Product Metrics: https://www.productanalytics.academy/courses/product-metrics?ref=22a3df
  2. Experiment Design: How to run product experiments until you get a statistically significant result https://www.productanalytics.academy/courses/experiment-design?ref=22a3df
  3. About Product Analytics Tools and how to choose/ use them as a PM: https://www.productanalytics.academy/courses/product-analytics-tools?ref=22a3df
  4. How to setup your product tracking plan: https://www.productanalytics.academy/courses/tracking-plans?ref=22a3df

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alaa MohyEldin

Alaa MohyEldin

Aspiring Product Manager, tech, and venture excite me, traveling around the world is my ultimate goal. https://www.linkedin.com/in/alaa-mohyeldin-97aa5880