Stop Analyzing, Start Thinking: What a Misguided Analysis at Amazon Can Teach Us

I’m a Senior Director of Analytics and I’ve been working in the tech space for 25 years. I’ve been leading analytics teams for the last 15 years across companies including Amazon, eBay, VMware, and more. Throughout my career I’ve uncovered many opportunities in the analytics domain to improve the quality of insights and the efficiency of organizations. Across all of the opportunities that I’ve found, there’s one that I feel is most frequently overlooked and has the highest potential reward — logical thinking.

This probably sounds absurd. How could logical thinking be missing at some of the biggest tech companies, and within analytically minded individuals? In a world with seemingly unlimited data and a rush to produce data for leadership requests, the thinking component is overlooked more than you may realize. Here’s one such example from my time at Amazon.

Amazon’s Member Churn Situation

Amazon has a membership program, Amazon Prime, which is the largest membership program in the world. Like any membership program, member retention is vital to the company’s success and to maintain revenue streams. But even with the best efforts, customers will inevitably churn. Given the importance of the membership program, it’s important to understand what causes customer churn, and then implement solutions to reduce customer churn. This all sounds straightforward on the surface, but it’s actually quite complex when you get into the details.

At Amazon, solving customer churn started off like it would at any other company which was by measuring the reasons for churn. Somewhere along the lines the product and leadership teams created a metric named, “Reasons for Churn”. A request was then made for the analytics teams to quantify the reasons into groups such as, “Customer Cancellation (through customer service)”, “Turning Auto-renew Off”, and “Billing Problems”, along with a few other reasons.

The analytics team, like many other analytics teams, did what they were asked to do. They pulled the data based on the request of their senior leader and delivered the results. They built a high-quality report and surfaced “insights” surrounding the number of churned members for each one of the churn reasons. While I’m not able to share the specific numbers, nor can I share how significant one reason was versus another, I’d like to talk about the investigation of “Billing Problems” for the sake of this article. I’d like to share how the churn analysis was off track before it even started.

The Problem with the Analysis

In Amazon’s environment, if a customer’s monthly billing for their subscription failed, their membership would be canceled and the customer would automatically be churned. When leaders saw the Reasons for Churn report and the line item for “Billing Problems”, they naturally asked for more data but this quickly led to deep dive analysis, the authoring of numerous multi-page documents, and more meetings than I can recall.

While a few minor issues were detected, the team never really seemed to make a significant of an impact on resolving member churn due to ”Billing Problems”. This wasn’t for lack of trying, but rather because they were focused on the wrong problem. These “Billing Problems” were never really a billing problem — at least not the type of problem that you’d imagine when you hear the words, “billing problem”.

At this point I feel like I may have you questioning my sanity given that I’m suggesting that there isn’t a billing problem when there’s literally a report for “Reasons for Churn”, with a line item and metrics for “Billing Problems”. But this is where the problem begins and there’s two things wrong with the analysis and approach.

A Flawed Metric

The metric for “Reasons for Churn” was a flawed and inaccurate metric. As mentioned, “Reasons for Churn” included categories such as “Customer Cancellation (through customer service)”, “Turning Auto-renew Off, and “Billing Problems”, along with a few other reasons. This is a flawed metric because those reasons aren’t actually reasons for churn. They are the means of churn, and there’s a big difference between these two concepts.

The reason for churn describes why a customer churns. The means of churn describes how they churn.

Unfortunately, the developers of the metric did not differentiate between the reason and the means. Additionally, they confused both of these meanings. Instead, what they actually needed to do was accurately define both “Means of Churn” as well as “Reasons for Churn“.

For the metric, “Means of Churn”, we would expect to include categories such as, “Billing Failure — Automatic Cancel”, “Customer Turning Off Auto-renewal”, “Customer Cancellation Request Through Customer Service”. For the metric, “Reasons for Churn” we would expect to include categories such as, “Unhappy with the Product or Service”, “Not Using the Service”, “Perceived Lack of Value for the Price”, and others.

However, by only creating a metric called “Reason for Churn”, and not considering “Means of Churn”, and not properly differentiating between these two metrics, product owners and leaders unintentionally solidified their belief that the actual problem was due to a billing problem. While it is possible that billing problems existed and were the reason for churn, this misunderstanding caused the team to overlook the possibility that a customer could have a reason to churn, such as being unhappy with their membership, but have a means of churn that results in a billing failure. Again, this is a very important distinction and where business context and business acumen become important in the analytics domain.

Flawed Approach

If we were to assume that the reason that a customer wants to cancel is due to unhappiness with their membership, we could then inquire about how (the means) they might churn. This inquiry and hypothesis surrounding the means of churn should likely lead to two ways to cancel that are easy to spot: Turing off their membership auto renewal feature, or calling customer service to request a cancellation. But there’s actually another way to cancel that isn’t as obvious. It’s also a means to cancel, which would cause a billing error.

There are actually a number of ways for this to occur. For example, a customer could instruct their bank to block the payment. The customer could also expire a virtual credit card (if used). They could also use a party service that allows for easy membership cancellation on the customer’s behalf.

By having this knowledge and business acumen, it’s clear how a customer could have a reason to cancel their membership, but not actually communicate their reason. Instead, their true reason could go undetected, and the means of cancellation would appear as a billing failure when the payment doesn’t go through.

The other flaw in the approach was that it wasn’t anchored to the customer. It was anchored to a process. By assuming that there was a process problem, the focus was around fixing that process. But with a focus around the customer, we could use some quick and logical thinking to prove or disprove that the majority of churn in this category was actually related to customer desires, and not a broken process.

Logical Thinking with a Focus on the Customer

In 2016, Jeff Bezos said, “We want Prime to be such a good value, you’d be irresponsible not to be a member”. If these churning customers related to “Billing problems” were receiving the value that Bezos envisioned, then we should expect the majority of those churned customers to resolve their billing problem and sign-up for their membership again with a different payment method. We should also expect to see a high volume of complaints about not being able to sign up or about the billing problem.

When I was at the company and I presented the shortcomings of the current metrics, my feedback was met with a fair amount of skepticism. But my logical question was this:

“If your mobile phone service was cut off right now due to a billing error, regardless of how that error happened, how quickly would you solve that issue?”

I think you can guess what that answer is. Immediately! Anyone who is realizing immense value from a product or service is not simply going to walk away when a technical problem, such as a payment processing error occurs. If we apply the same logic to the customers who churned from their membership, for the supposed reasons of “Billing Problems”, then we should expect almost all of these customers to sign up again.

This isn’t to say that there couldn’t be any billing problems. But by leveraging this type of thinking, analysis shortcuts can be taken to avoid going down rabbit holes and to ensure that teams are focused on the right problem.

Conclusion

If you want to solve the most challenging problems that your organization is facing, you must resist the urge to immediately dive in and start pulling data upon request. Instead, here’s what I would recommend.

First, ground your analysis in a thorough and thoughtful understanding of your products, business processes, and your customers mindset and behaviors. Second, realize that all of the information that you need to solve the problem might not be inside of your database. You’ll likely need to rely on logical thinking and strong business acumen. Third, you’ll need to ensure that your metrics are properly defined and you need to ensure that those metrics don’t accidentally cause a misinterpretation of what’s actually occurring within your business. And last, it’s crucial that you leverage logical thinking across each problem that you encounter.

By following these steps, you’ll be able to take shortcuts with your analysis and avoid wasting time and effort investigating items that may not lead you to the root cause. Also, you’ll avoid coming up empty handed in your analysis.

Brandon Southern, MBA, is a Sr. Director of Analytics and the founder of Analytics Mentor. With a 25-year career in the tech industry, Brandon has excelled in diverse roles encompassing analytics, software development, project management, and more. He has led analytics teams at Amazon, eBay, VMware, GameStop, and more.

You can learn more about Brandon and Analytics Mentor at http://www.analyticsmentor.io/