Monitoring Metrics Puts You in the Driver's Seat

Edward Deming, the quality guru, says, 'You cannot manage what you do not measure.' Mature businesses rely on metrics to gauge product performance in the market as well as monitor the cost of building the product.

Product managers in technology companies typically don't spend enough time analyzing product metrics. The numbers are sometimes hard to obtain and the volume of tactical activities they are faced with is overwhelming. When they do manage to carve out the time to figure out what's next, they don't know where to begin. As companies downsize to bring expenses under control, more work piles up on the already overworked product managers who remain. And yet, executives expect product managers to run their products like a business.

In the fast-paced world of technology, product managers need to balance time-to-market pressures with risk mitigation. Market analysis, which takes time to do, is simply done to mitigate risk: risk that you're building something no one wants; risk that you can't deliver something to the market profitably.

In mathematical terms, there are infinite things to measure and analyze. But, in the world of practical product management, we can't measure everything. We need to focus on a few key performance indicators (KPI's) that give us feedback that we are on track or indication that we need to make changes before it is too late. The question is 'What should I measure and analyze?'

Think about your car. It has gauges and dials and displays. They provide measurements or metrics. But which gauges do you actually need to understand and monitor? For the novice driver, the speedometer is the first gauge you learn about. How fast are you going? You compare that to the speed limit. As you drive, you monitor your speed and either accelerate or decelerate based on external conditions. Another gauge a novice must understand is the gas gauge. How far can you travel before you run out of gas? You learn to plan a trip to the gas station to fill up the tank before it is empty. The odometer tells you how far you are traveling.

As you become more proficient, you start paying attention to other metrics such as oil level and tire pressure. A car buff or professional race car driver cares about oil pressure and revolutions per minute (RPM). When you take your car in for routine service, your mechanic will check the alignment, the timing, fluid levels, and will probably perform a 20-point computer-based analysis of other 'systems? in your car. But normal drivers don't need to know about these when driving around town.

Just as we need gauges and metrics for operating a car, we need similar metrics to run a product. This article highlights ten product metrics that product managers should monitor. Some gauges are harder to find or see than others, but all provide insight into either how well the product is performing in the market or what impact the product has on the organization. The focus of this article is on business-to-business technology products sold primarily through direct sales channels but can be applied to any technology product.

For any metric, you should first establish a baseline measurement. Then, keep track by period (weekly, monthly, quarterly, or annually) and observe the trends. Establish goals and measure how you are doing. If you are not meeting the goal, find out why. And remember, the answer to the question is probably not in the building. Leave the building and interview customers and potential customers to find out what is going on. Once you have figured out what is going on, you will need a plan or a program to improve or change the situation.

The first one you should look at is the customer gauge.

New/lost customers

How many new customers are you acquiring per month? How many customers have canceled maintenance or subscriptions per month?

These metrics tell you whether your customer base is growing or shrinking. You need both of these numbers. You might have high customer retention (few customers ever leave you), but if you are not also acquiring new customers, eventually you might not have enough business to sustain the company. Customer count is the speedometer of a product. Falling customer numbers indicate that the product will fail. This information should be easily attained from sales, customer service/technical support, or accounting. A more advanced customer metric is to calculate the lifetime value of a customer. How much profit does each customer generate? Go to accounting for this. It is an important metric, but more like the computer-generated analysis your mechanic does.

Product quality

Product quality should be monitored on an ongoing basis to make sure it doesn't become a serious problem. Set up a spreadsheet with either weekly or monthly statistics of the number of defects. A stacked bar chart can show you trends over time.

For released products, technical support typically keeps track of open defects. As soon as a product is released, it is common for the number of defects to go up as more people use the product. If the quality is high, it should be a manageable number and should settle down after awhile. For unreleased products, ask quality assurance for the statistics. If the number of open defects is not going down as the release date approaches, the release date is likely to slip. Obviously we want the Sev 1 and Sev 2 defects approaching zero as the release date approaches. (Releasing a product with poor quality will cost you money, time, and reputation--don't even go there!) Severity four 'defects' are typically enhancement requests and are often ignored in terms of product quality. Yet they indicate areas where the product fails to satisfy the customer. Product quality is the best indicator of internal health of the product; quality is the gas gauge of the product.

Technical support

Technical (or Customer) Support already measures the number of calls per product and the nature of the calls. Product management should analyze these numbers to identify areas of that product that can be improved for a better customer experience. Are there problems with installation of the product? Is the documentation too hard to understand? Is the product too hard to use? By streamlining a process, will you cut down the number of technical calls? Review the number and type of calls for your product to uncover hidden profit leaks.