It has been said that what we think as product managers, while interesting, is irrelevant. Any product manager who has gone toe-to-toe with a development, sales or marketing manager knows this to be true; they're generally not willing to bet their badges on something they worry we may have just made up. And we’ve all had the experience of bringing a requirement to our development or management teams, having them ask “where did that come from”, and being caught short on evidence. It’s not a happy feeling.

Taking that negative and turning it into a positive, I’ll offer it’s not what we think as product managers that matters, it’s what we know. For product managers knowing is defined as being in possession of quantitative and qualitative information that can be supported by evidence a third party can examine. Our job is largely built around acquiring this information and translating it into intelligence that the organization can use to build things people want to use and buy.

It can be argued that this process of knowing is at the very heart of the product management discipline. When we gather market requirements and dig up competitive intelligence, when we ask people to describe their problems and preferences, we’re working to know. All of the basic product management processes and documentation relating to strategy, roadmaps, requirements and release definitions are designed to take the task of building products out of the subjective space of opinion and anchor it in the objective reality of a knowable, repeatable process. When this process is fueled by evidence that can be examined and known we increase the odds of a successful outcome.

And yet there’s a nagging, universal tension between our "desire to know" and our "need to act". This tension is supposed to be addressed by performing a risk analysis in which the costs associated with becoming knowledgeable are weighed against the risks associated with acting without knowledge. This decision risk threshold varies by organization, by product, by project and by person, and obliges the product manager to constantly calibrate the decision process to the situation.

Now this sounds good on paper – in practice it is much more challenging, especially with regards to knowledge the organization doesn't generate itself (e.g., market metrics, buyer/user perceptions). Unless the organization is in immediate possession of evidence of what its target market does/wants/believes/thinks, it’s going to take time and money to gather the information needed to “know”, and the sad reality is that often product managers don’t have the time to go find out. On the cost front, even if product management has the time, the best sources of information or ways of acquiring it are often expensive. Analyst reports, market research projects, travel, focus groups and prototype development aren’t free, and few product managers have a dedicated research budget they can direct.

Then there is the question of “how much knowledge is good enough”: we spend time with customers and prospects asking good questions to uncover market problems, but how can we be sure that we’ve talked to enough of them? We sit in on beta calls and ask questions until we’re blue in the face, but how can we be confident that we’ve gotten enough reliable feedback to move ahead? How many focus groups do we need to hold? How many analysts do we need to talk to?

The answer is generally “when you start hearing the same thing over and over again.” Except when you don’t – which is often the case with research into new markets. Scott Anthony, the President of Clayton Christensen's innovation consulting firm Innosight, describes.

"How can you describe performance dimensions the customer can't imagine? How can customers project usage of something they have never experienced? As the old saying goes, "Markets that don't exist can't be measured and analyzed."

The question of when and where to use quantitative research versus qualitative research has been summarized as “if you want to know what people think and do, choose quant; if you want to know why they think and behave that way, choose qual”. For the product manager, the question isn’t so much “which sort of research is more appropriate” but rather “do I have any research at all I can point to”.

At the highest level, consider that most organizations are set up to capture, track and report on key performance indicators relating to their products, and they use this data to advise many different types of decisions. But how many of these indicators are “mashed up” with market information? For example, it is easy to track how quickly revenue grows post-launch. But how many organizations know whether that success is based on a shallow usage of the product, which leaves the company exposed to competition from utilities? Or when considering roadmap options, how many organizations are able to point to market evidence that would allow them to tune platform support choices to match what customers are really using, as opposed to what vendors report?

If we agree that it is essential to know, and we acknowledge the cost of knowing is often high, then we need to find ways to reduce the cost of knowing if we're going to be able to do our jobs as product managers. So what's stopping us?

I’ve got a few ideas why. Later on I’ll bring all of these thoughts together with some concrete examples.

1500 years ago, everybody knew that the Earth was the center of the universe. 500 years ago, everybody knew that the Earth was flat. And 15 minutes ago, you knew that humans were alone on this planet. Imagine what you'll know tomorrow. -- Agent K to Agent J, Men In Black

With the arrival of cheap online tools for gathering market intelligence, the growing adoption of CRM systems to track customer feedback, and the vast reach of the web as a resource for research, product managers find themselves awash in more data than they’ve ever had access to. So why are we still talking about this problem of "knowing"?

Consider this scenario - development leaders responsible for a SaaS application analyze daily usage data, document what users doing and using, and associate “what is used” to “what is valued”. As a result, they are able to justify all of the items on their proposed enhancement and fix list as “necessary to support our customers”.

In this situation, someone is following through on a correct impulse - to gather and use objective data in an effort to understand the customer. But does the data flowing from the application reveal anything about the problems that exist beyond it? Are the users the sole determinants of value - in other words, does making them happy equate to real competitive advantage and more sales?

And consider this second scenario - a CEO goes on a whirlwind multi-city tour to “go hear what the marketplace is saying”. After visiting with anywhere from 25 to 40 different prospects and customers over a two week period, the CEO returns with “voice of the customer” conclusions and delivers this information to product management along with instructions to incorporate the feedback into the 12-month roadmap.

In this situation someone is also following through on a correct impulse - to develop a subjective understanding of the needs of the customer and the marketplace. But does the data that flows from this inquiry allow the product manager to really know what is going on in the market? Were those the right people to go talk to? Were the questions asked correctly? Does the feedback contradict other “more scientific” market intelligence that’s already been used for planning? And who is going to tell the CEO that what was “learned”, while interesting, isn’t necessarily actionable?

The reasons product managers often don't explore these questions is because they take time and money to answer correctly - and because one person's one-off methodology for gathering market data is as good as anyone else's, just as it's hard to argue in the face of volumes of objective "evidence". In this case, the reliability of the original data is "good enough" to be actionable; decisions are made, and life goes on. This second scenario is especially challenging because the data came from higher up in the organization. The pressure to go along with such conclusions can be overwhelming.

All too often irregular or limited market research becomes organizational gospel, and is used to justify decisions for much longer than is prudent. This results in a perception that the cost of knowing more exceeds the value of knowing more. "We have better things to do with our time and money than go out to confirm what we already know" is something we've all heard. “We’re already paying a fortune for [insert analyst firm] research – isn’t that good enough?” is another.

The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them. – Dr. W. Edwards Deming, The New Economics for Industry, Government, Education

As is true with so many challenges, my experience has been that the way to address this one is to take a systematic approach to fixing the product development process from the start. If certain key objective and subjective metrics are required before meaningful development can start, then the product manager can avoid the sort of ad-hoc flurries of investigation that everyone finds so frustrating. In effect, I’m recommending a “we have to know x before we can move ahead” test.

Management can still make explicit decisions to move ahead without x - this “we’re moving ahead without certain data” decision is essential to making the process work, as it calls attention to gaps in knowledge that introduce risk, and justifies the potential acquisition of additional intelligence in the future to mitigate that risk.

Because all good ideas need a catch-phrase to make them stick, I’ll refer to this process as “instrumenting the marketplace”.

The decision to instrument the marketplace is not one that can be made lightly, as it carries with it a commitment to spend time and money to achieve it. To justify the expense, the product manager must lay out a portfolio of data that is considered valuable – then the negotiation starts.

In this process, it’s important to be creative; there are many sources of market, competitive and runtime intelligence data now available to software development organizations that have traditionally been difficult or even impossible to acquire without significant expense or customer impact. This is one of those discussions that benefits from a great big round-table discussion where everyone on the team can engage their individual creative talents.

Here’s my explicit call-to-action manifesto, followed by a breakdown of each important element. I know this is high-level stuff, but I hope it will make sense when I bring in my examples:

To be successful over time, product organizations should instrument the marketplaces they serve by building, buying or partnering to get access to timely, accurate, flexible, reliable, and scalable systems that extract objective and subjective data from those marketplaces, and should make that data a required element of the regular business intelligence process.

Instrument the marketplace

The concept of instrumentation is familiar to everyone in the software business. What instrumentation does is identify the atomic events, discrete event intervals and relevant environmental data we wish to know more about, and establishes a pipeline for delivering that information back to the company. When we apply the idea of instrumentation to the marketplace, we're forced to consider what can and can't be known in a way that can be reliably detected and measured over time. We're obliged to define what we need to make decisions. This very act of choosing what we want to know is extremely powerful - it forces us to agree on the evidence we require to propel our decision processes. It also obliges us to have an agreed-upon strategy in hand before we begin, otherwise we're left with a "tail wagging the dog" scenario of making strategies dependent on measures.

Build, buy or partner

Some of the ways in which a product organization instruments the marketplace are proprietary; Microsoft's Customer Experience Improvement Program is one example (see below). Other ways are clearly available to everyone; analyst reports come to mind. A third requires a combination of third-party technologies and competencies that are augmented by the unique needs and know-how of the software organizations using them; software instrumentation technologies and focus group specialists are examples. The reasoning is that all options for instrumenting the marketplace should be considered, not just one.

Timely

What is true with bread and eggs is just as true with information. The market insights and competitive data that were so fresh a year ago when you got them may be interesting right now from an historical perspective, but their usefulness isn’t what it used to be. In fact, it is often more dangerous to make decisions supported by old data than it is with no data. When instrumenting the marketplace, evaluate how often certain data types and sources are refreshed, and work to obtain a mix of data covering different time periods. Modern software deployment tools make it possible to obtain real-time or near real-time views of user activity, application performance and quality, which when combined with other feeds can give an organization greater clarity into the marketplace than ever before.

A note of caution – the freshest data is often the most expensive. Calibrating what you think you need versus what you use should be an ongoing exercise.

Accurate

This is the most challenging of all of the five characteristics of market instrumentation data. Because of the Hawthorne Effect (people will change their behavior when they know they are being observed), there is an impression that certain market research data is inherently unreliable, especially focus groups or small-scale 1-to-1 interactions. For this reason any efforts to instrument the marketplace should begin with a careful analysis of whether the data obtained from each activity can be either trusted on its own merits or in the context of other trusted data. Note that I'm focusing on accuracy, not precision - for the large volumes of data required by a functioning marketplace instrumentation project, precision isn't required.

Flexible

One reason the cost of knowing is perceived as high is because market research has a bad reputation as a "big bang" event - so much time is spent formulating questions, designing studies, conducting research, gathering the data and interpreting it that by the time you get the results you wish you had asked different questions. Instrumenting the marketplace, by comparison, presumes there is a steady stream of data flowing back to the company that can be used to answer a variety of questions, so that you can be prepared to address a variety of research requirements without the need to go back to the beginning.

Reliable

There are few things more frustrating than a gap in your intelligence caused by an interruption in the flow of data. A reliable system for instrumenting the marketplace should be robust enough to deliver streams of data without interruption. This is especially valuable if you require volumes of historical data to validate questions you haven't anticipated – because certain events, left unobserved, can’t be reproduced reliably or objectively down the road.

Scalable

Very often the organization's "need to know" starts with one product, typically during the initial requirements process. A scalable approach to instrumenting the marketplace allows the software organization to begin with a narrow set of data requirements and expand them as needed over time. An example would be working with a market research firm that specializes in all of the markets you serve, but only focusing them on a single product at the outset. Another would be considering a software instrumentation technology that can be expanded over time to cover more products and different features without the need for fundamental architectural changes to the instrumented products or the reporting infrastructure.

To extract objective and subjective data

It is important to acknowledge, as Dr. Deming did, that not everything can be definitively "known". We can count how many copies of our software are purchased, but can we measure how often a specific feature on that product is used, and why? We can track the activities of our competitors in the marketplace, but can we measure why our customers are choosing their products over ours, or vice versa? The conclusion is that to "know" our customers in certain ways we have to talk to them; to anticipate how changes in the world they work in will impact their behavior and attitudes toward our company, our solutions, and our industry, we need to give them an opportunity to describe it to us. When combined with objective data, this subjective data will animate the objective data and allow us to draw conclusions from it that may not have been possible from analysis of the objective data alone. The goal here is to avoid relying exclusively on facts or impressions, but to make a practice of using both.

Make that data a required element of the regular business intelligence process

It is ironic that most software companies have elegant and sophisticated systems for managing their financial operations but only the most rudimentary systems for managing the product definition process. Because of the inherent limitations associated with traditional market intelligence data types, and the risks this creates for managers who orient themselves around that data, most organizations fail to include them in the normal business intelligence process. When the data available from instrumenting the marketplace can be demonstrably accurate, flexible, reliable and scalable, it becomes as trustable as other sources of data the company produces and analyzes regularly.

Trust, in this case, is the key. In the absence of trusted data, all market assessment activities devolve to subjective exercises in which the most senior, most vocal or most persuasive argument wins. With trusted data, all parties regardless of their individual opinions are obliged to consider the same evidence. In such an environment, the product manager has a system in which the value of "knowing" exceeds its cost, because decisions are not made without it.

The enemy of inquiry is authoritarian certainty. – M. Robert Gardner

Evidence of software development organizations’ efforts to instrument their marketplaces can be found by looking for streams of information that match the qualities I described above. The most visible examples tend to be feedback mechanisms that vendors build into their software products.

Microsoft has a long history of using a wide variety of market research techniques to advise their development efforts, but these don’t scale well when it comes to gathering meaningful feedback from the hundreds of millions of users of their software products. Without such a scalable system, any effort to instrument their marketplace would deliver only partial value.

To address this gap, a number of years ago Microsoft created their Customer Experience Improvement Program (CEIP), an opt-in software instrumentation solution which they have built into many of their products to stream intelligence on application runtime behaviors back to the company. They describe the CEIP in the following way:

CEIP collects information about how [Microsoft] customers use Microsoft programs and about some of the problems they encounter. Microsoft uses this information to improve the products and features customers use most often and to help solve problems.

By using this information both proactively and reactively, Microsoft is able to be market-aware in ways that they could not if they had to rely exclusively on traditional market research techniques. With knowledge of how their users experience Microsoft software, they are able to “give all Microsoft customers the ability to contribute to the design and development of Microsoft products”.

An example of a smaller company with similar challenges is K2. In a published case study they describe a problem they faced with obtaining sufficient market intelligence during the beta process. They reported a 10% response rate from beta participants when traditional feedback mechanisms such as email request, feature and bug tickets, form postings and phone calls were used, and determined that this was not sufficient data on which to base their decision process.

Their approach to addressing this issue is similar to Microsoft’s, in that they elected to embed an automated way for their applications to stream intelligence about their application’s runtime behavior back to the company. The difference was that rather than build the solution from the ground up, they opted to employ an available off-the-shelf solution. The CEO of K2 described the impact of this as helping them “focus our product planning, development, and QA resources on shipping only the very best software possible.”

What is telling to me about both of these examples is that the information gathered from these programs can be used in more ways than can be imagined – both proactively and reactively. The challenge is clearly taking the first step to identify what is needed, and implement systems for obtaining it.

There are very real barriers to implementing such systems. Issues of data privacy, availability, ownership and the cost of creating and maintaining them must be put on the table before such a project can start. Often the very last thing on the mind of development is to implement instrumentation - it's the job of product management to make the case for why it should be one of the first.

Quis custodiet ipsos custodes? – Juvenal

If the product manager is supposed to be the messenger of the marketplace, then the credibility of the process the product manager uses to acquire this information “watches the watchman”. Given the potential costs associated with acquiring data, product managers must explore ways to access automated, optimized “streams” of appropriate data that can be used to perform ongoing assessments of the marketplace. The creation of processes that "instrument the marketplace" is the best way to repeatedly and reliably acquire data that can be trusted enough to incorporate into the regular business intelligence processes of the company. Once incorporated into these business intelligence processes, the value of knowing grows to far exceed its cost.

With experience in sales, field engineering, communications, marketing and product management, Bob Corrigan knows how to build products that people want to buy - and how to get them sold. He has held senior product management roles at large and small companies in a variety of industries when he's not working as a voice actor or piano player. He blogs about product management, marketing and whatever else he's thinking about at acknak.blogspot.com, and can be contacted at bob.corrigan@gmail.com.