Most good product managers learn this lesson relatively early in their career, but it’s an important one and worthy of a blog post. The subject is predicated on a concept from statistics; namely, the larger the sample size is from a given population, the more representative the sample is of the entire population and a concept from psychology, called “confirmation bias.”
Talking with customers about their pain with your product is an excellent discipline. These conversations provide a product manager with impactful anecdotes that a PM can internalize and spread within his or her organization. But a PM needs to understand the inherent flaw with only speaking 1:1 with customers.
When you’ve spoken directly with a good sample of customers, you generally have awareness of customer requests, but – and this is critical — you don’t have an understanding to the extent to which the requests are important, or how strongly they are affecting your total customer base. Generally speaking, it is difficult to talk with enough customers to understand how you should focus a very limited resource — in the case of software development — engineering time and brainpower.
Of course, there are exceptions. If you meet with 20 customers and they all have the same product pain, your job is done. But that is usually not the case. Usually, you talk with 20 customers and receive 20 different feature requests or product quibbles.
Furthermore, product managers are human and, like everyone, are subject to a phenomenon from psychology called “confirmation bias.” Let me know if this scenario sounds familiar: You have a gut feeling that everyone wants Feature X. You find one or two customers that want Feature X, and you unconsciously inflate the extent to which this need is requested by the user base. It happens — you need to be aware of the bias and work to overcome it.
So what’s the cure? True perspicacity into the needs of a user base is the result of talking with customers, and complementing that direct knowledge with data from a larger sample. Qualitative research alone gives you direction and awareness of issues – and I fully admit that awareness is 70% of the solution. But quantitative research gives you conviction regarding the extent of issue.
With both perspectives, you have a better understanding of how deeply to apply limited resources to a problem, as well as minimize your personal bias.
We’ll save the topic of actually executing a valuable and statistically valid quantitative study for another blog post.
thought provoking post, Jon.
quant is one way to handle it… the other is to just go with your gut & gauge customer reaction.
in startups, i prefer the latter. trying, failing, and rebuilding is a much better strategy IMO when agility is your competitive advantage. as you know
larger companies have huge resource advantages but they are slow. they can take a long time to just figure out what the right question is, much less performing a quant study to verify a hypothesis. and by that time the study is no longer valid because market dynamics especially on the Web shift so fast.
i would be very interested to see your thoughts about performing quant studies effectively in a startup environment. how can quant supplement the agility & speed of “doing” that makes better startups so effective. a combination of the two impresses me as the way to go.
Chris