Avoiding Vanity Metrics

Early in my career in the analytics field I started to become inundated with task request from stakeholders.  I felt overwhelmed and confused at times by what my stakeholders were asking for.  There were already countless dashboards and metrics, yet I was receiving more request to produce additional metrics and add another metric to a dashboard or report.  Out of frustration with requests that didn’t seem to add value and only seemed to clutter my dashboards and create busy work, I decided to push back on my stakeholders. I chose to become curious, ask questions, and push for more information when the request didn’t seem to make sense.

In the short term this created some frustration for my stakeholders as they weren’t used to being challenged. But over the long term they became used to this process.  What resulted was the avoidance of the creation of metrics that I refer to as vanity metrics.

Vanity metrics are metrics that sound interesting or make you feel good, but they don’t provide any actionable insights or value to the business.  The tricky thing about these metrics is that they can be hard to identify because every metric can be useful when applied to the right business context.  My peloton bike has a prime example of a vanity metric called Number of Rides.

Example of a Vanity Metric

Nearly every morning I hop on my Peloton bike, log into the app, and ride along with one of my favorite instructors.  During the ride my instructor exclaimed (to another rider), “Congratulation! Wow! 1,000 rides! That’s amazing!”. As impressed as my instructor is, the number or rides is something that I would consider to be a vanity metric.

On the surface, 1,000 rides seem impressive.  But if we deep dive into this metric to explore how it is calculated, what we find is that whenever a user completes a ride, the total Number of Rides increases by 1.  This sounds logical.  But if we look deeper, we find that Peloton offers 9 different ride durations, ranging from 5 minutes to 90 minutes.  This means that when riders are being congratulated for their impressive number of rides, my Peloton instructor isn’t comparing apples to apples across riders.

For example, let’s assume that Rider-A rode 200 rides and Rider-Z rode 1,000 rides.  On the surface we may assume that Rider-Z has been a long-time rider, rides multiple times per day, or is more consistent than Rider-A.  But this may be far from the truth.  If Rider-A rode 200 rides but took the 90-minute class for each ride, and Rider-Z rode 1,000 rides but took the 5-minute class for each ride, a different metric would make us feel different about which rider is more impressive.

When we calculate a new metric based on the number of minutes, Rider-A rode 18,000 minutes and Rider-Z rode 5,000 minutes.  Suddenly, Rider-Z is nowhere near as impressive of a rider than Rider-A.  It is for this reason that I consider the Number of Rides to be a vanity metric.  It doesn’t offer a fair comparison of the rider activities, nor does it help the business to derive any actionable insights.  But there’s a catch.

While I may perceive the number of rides may be a vanity metric, we must consider the business context.  If the goal of surfacing the metric is make someone feel good, then the metric is probably a great metric and ironically, is a vanity metric, but in the sense spending 40 minutes taking the perfect selfie next to a Lamborghini. 

When is it Not a Vanity Metric?

If Peloton is researching the number of new classes, it would be important to know the number of rides completed based on the class duration.  However, this is slightly different than describing the number of rides for a specific user.  But we could find a use for this metric in a situation where we are trying to understand user preferences.

By understanding the number or rides for a user, this may lead us to understand that Rider-Z finds value using her Peloton, and that she uses it daily.  And in our exploration of the data, we may find that Rider-Z takes 3 classes every day, but the classes are only 5 minutes in duration.  Given this information, Peloton may decide to produce more 5-minute classes so riders such as Rider-Z will frequently have new content to view.

However, this is no longer the same metric.  We’ve now produced a new metric called, Number of Daily Rides per Rider, or we may produce a metric called, Number of Rides per Day, per Rider, per Duration Type.  These metrics serve a specific business purpose and are different from the original metric of Total Number of Rides.

Context and a Better Starting Point

What this means is, context matters. If we don’t understand the business context, then we may be at risk for producing metrics that don’t add any value to the business. Sure, they may be interesting, but if they aren’t actionable, we’re creating busy work, cluttering our reports and dashboards, putting ourselves at risk of introducing bugs, creating technical debt, and districting our focus from something that is more valuable to the business.  To avoid creating vanity metrics, I recommend working backwards as opposed to simply asking for a metric. 

First, understand the business, customer journeys, user flows, operations, and anything else that is related to your domain.  Second, describe a current or potential situation that you’d like to understand, for the purpose of improving a situation or avoiding a situation.  Last, determine the best means of describing the current state of the situation by using the proper metric.

By working backwards from the business situation, you can determine the best metric to use.  And by displaying the right metric, you’ll add valuable insights to the business while avoiding clutter and confusion. And you’ll stay focused on the business problem.  It’s okay to be curious, but it’s important to be focused and deliberate when creating new metrics for your business.