The product management and startup worlds are buzzing about the importance of "validation". In this entry, I'll explain how this idea originated and why it's leading organizations astray.
Why Validate?
In lean startup circles, you constantly hear about "validated learning" and "validating" product ideas:
The assumption is that you have a great product idea and seek validation from customers before expending vast resources to build and bring it to market.
Indeed, it makes sense to transcend conventional approaches to making product decisions. Intuition, sales anecdotes, feature requests from customers, backward industry thinking, and spreadsheets don't form the basis for sound product decisions. Incorporating lean startup concepts, and a more scientific approach to learning markets, is undoubtedly a sounder approach.
Moreover, in larger organizations, sometimes further in the product life-cycle, everyone seems to have an opinion about such aspects of the business model as:
Validation in Practice
But let's consider what our naive lean startup practitioner (we'll call her "Cameron") does in practice.
Present the idea
Cameron puts together a slide deck, minimum viable product (MVP), or demo, presents her idea to one or more prospects, and eagerly awaits feedback. The prospects say, "I like it!" or "I would buy it!" Cameron feels warm inside, as the prospects have "validated" her and her idea.
Ask about the "pain point"
Since solving an urgent and pervasive problem or "pain point" is usually a prerequisite for product success, Cameron visits with prospects and asks them if they experience one of the problems she envisions her product would solve. "Yes," reply the prospects. Feeling "validated", Cameron does an internal high-five with herself.
Name a price
Cameron asks prospects what they would pay for a solution with the features she envisions for her product. Prospects "name their price". Or, in equally naive fashion, Cameron herself names a price and asks, "Would you pay X for the product?" Prospects reply affirmatively. Cameron can hardly contain her giddyness as her pricing assumptions are "validated".
Cameron feels extra special for having "gotten out of the building" to visit customers.
What's Naive about Cameron's Approach?
The results of all of Cameron's efforts are practically worthless, aside from the emotional affirmation she may feel. When you visit prospects, you don't get reliable information by posing hypothetical questions. When you seek validation from someone, you will tend to get it. Cameron unwittingly designed her interactions with prospects in a manner likely to confirm her preconceived ideas, and her interpretations of the results were naive. Let's call it "validation bias", an insidious psychological disorder that has infected the lean startup community.
Note: had Cameron conducted a survey, though seductive because it would have yielded quantitative data, the results would still have been suspect to the extent the questions were hypothetical and failed to confront prospects with real choices and commitments.
What's the Alternative?
If you want to be scientific in your approach to product decisions, you craft experiments and make falsifiable predictions, with the intention of testing, not "validating", the hypotheses and underlying assumptions.
Philosopher Karl Popper is famous for having championed falsificationism, a set of principles that shifted the emphasis from verifying scientific beliefs to ensuring they are possible to falsify via experiments:
According to Wikipedia:
What Cameron Could Do Differently
Instead of asking hypothetical questions, Cameron could ask the prospects what they actually do (and have done) instead of what they would do. She could probe into their current and past experiences and note whether the supposed problems manifested themselves in those experiences. She could sit with prospects in their native environments and observe (ethnography), first hand, their situations and challenges. Cameron could test her value proposition and pricing hypotheses by quantifying the costs that problems and their workarounds impose on prospects. She could ask for an actual commitment to pay for the product once it's released.
Cameron could also work with her team to craft "digital" experiments and predict how those experiments will turn out. For example, her team could author "how-to" guides, downloadable on landing pages, that help prospects solve the problems she assumes they face. The team could predict how many people will visit those landing pages and how many people will proceed to download the guides. (To make prospects aware of the how-to guides, the team could run Facebook ads or Google AdWords to drive people to those landing pages and see if the predicted number of page visits and downloads materializes.)
The possibilities are endless, but they require a mindset that emphasizes falsifying product ideas and business model hypotheses, not "validating" them.
Why Validate?
In lean startup circles, you constantly hear about "validated learning" and "validating" product ideas:
The assumption is that you have a great product idea and seek validation from customers before expending vast resources to build and bring it to market.
Indeed, it makes sense to transcend conventional approaches to making product decisions. Intuition, sales anecdotes, feature requests from customers, backward industry thinking, and spreadsheets don't form the basis for sound product decisions. Incorporating lean startup concepts, and a more scientific approach to learning markets, is undoubtedly a sounder approach.
Moreover, in larger organizations, sometimes further in the product life-cycle, everyone seems to have an opinion about such aspects of the business model as:
- The most pervasive and urgent market problems the product should solve.
- Whether the solution truly addresses the market problems.
- What price customers would be willing to pay.
- What tactics will most effectively drive prospects to buy.
Validation in Practice
But let's consider what our naive lean startup practitioner (we'll call her "Cameron") does in practice.
Present the idea
Cameron puts together a slide deck, minimum viable product (MVP), or demo, presents her idea to one or more prospects, and eagerly awaits feedback. The prospects say, "I like it!" or "I would buy it!" Cameron feels warm inside, as the prospects have "validated" her and her idea.
Ask about the "pain point"
Since solving an urgent and pervasive problem or "pain point" is usually a prerequisite for product success, Cameron visits with prospects and asks them if they experience one of the problems she envisions her product would solve. "Yes," reply the prospects. Feeling "validated", Cameron does an internal high-five with herself.
Name a price
Cameron asks prospects what they would pay for a solution with the features she envisions for her product. Prospects "name their price". Or, in equally naive fashion, Cameron herself names a price and asks, "Would you pay X for the product?" Prospects reply affirmatively. Cameron can hardly contain her giddyness as her pricing assumptions are "validated".
Cameron feels extra special for having "gotten out of the building" to visit customers.
What's Naive about Cameron's Approach?
The results of all of Cameron's efforts are practically worthless, aside from the emotional affirmation she may feel. When you visit prospects, you don't get reliable information by posing hypothetical questions. When you seek validation from someone, you will tend to get it. Cameron unwittingly designed her interactions with prospects in a manner likely to confirm her preconceived ideas, and her interpretations of the results were naive. Let's call it "validation bias", an insidious psychological disorder that has infected the lean startup community.
Note: had Cameron conducted a survey, though seductive because it would have yielded quantitative data, the results would still have been suspect to the extent the questions were hypothetical and failed to confront prospects with real choices and commitments.
What's the Alternative?
If you want to be scientific in your approach to product decisions, you craft experiments and make falsifiable predictions, with the intention of testing, not "validating", the hypotheses and underlying assumptions.
According to Wikipedia:
"A statement is called falsifiable if it is possible to conceive an observation or an argument which proves the statement in question to be false."In fact, Popper argued that a statement isn't scientifically factual at all if it isn't falsifiable.
What Cameron Could Do Differently
Instead of asking hypothetical questions, Cameron could ask the prospects what they actually do (and have done) instead of what they would do. She could probe into their current and past experiences and note whether the supposed problems manifested themselves in those experiences. She could sit with prospects in their native environments and observe (ethnography), first hand, their situations and challenges. Cameron could test her value proposition and pricing hypotheses by quantifying the costs that problems and their workarounds impose on prospects. She could ask for an actual commitment to pay for the product once it's released.
Cameron could also work with her team to craft "digital" experiments and predict how those experiments will turn out. For example, her team could author "how-to" guides, downloadable on landing pages, that help prospects solve the problems she assumes they face. The team could predict how many people will visit those landing pages and how many people will proceed to download the guides. (To make prospects aware of the how-to guides, the team could run Facebook ads or Google AdWords to drive people to those landing pages and see if the predicted number of page visits and downloads materializes.)
The possibilities are endless, but they require a mindset that emphasizes falsifying product ideas and business model hypotheses, not "validating" them.
Comments
To conduct interviews and experiments that help us truly test our business model, we have to ensure we're not biasing the responses.
When a prospect is sitting with you, and you ask her if she likes your idea, it's your idea. If she says she doesn't like it, she'll in many cases be uncomfortable trashing your idea.
If you ask if the your assumptions regarding the problems she faces are valid, she'll similarly be uncomfortable telling you you're wrong.
Moreover, prospects aren't experts on product ideas and business models. They're experts on what the have done, and what they currently do.
So you need to craft your questions in a way that doesn't bias responses, and you have to focus on actual experiences, not hypotheticals.
I wrote about many of these issues in my entry on the top mistakes product managers make when conducting prospect interviews.
Anyways LS clearly says experiments must be falsifiable.
For example problem interview must ask provide an answer to questions like
Do you have the specific problem I am trying to solve. Answer is either yes or no. An objective measurable response.
How do you solve the problem currently, confirms the alternatives. Again very precise.
And so on. Its true to making experiments which are falsifiable is heart breaking perhaps for entrepreneurs but then not forming is their fault not of the methodology.
However, neither of the sample questions you posed is likely to yield reliable information. While it's possible that the prospect will answer "no", they may be hesitant to let you down with a negative answer.
Instead, I recommend asking open-ended questions about what prospects actually do, not posing your assumptions and eagerly (and obviously) awaiting a positive response.
After reading the LeanSU manual front to back it's clear to me how many entrepreneurs haven't bothered to read the materials. Those that have, how many take it as a cast iron blueprint to success whilst jumping on the hype bandwagon after attending a few seminars/workshop surrounded by positive high fives. Reminds me of the Tony Robbins self help crew.
I hear these terms like lean SU, validated learning, actionable metrics knowing that most have very little deep scientific understanding about implementation or why it's so vital.
The problem with many entrepreneurs is themselves which they never factor in their equation, hence the absolute need for rigorous scientific analysis.
As Elon Musk explains "natural human tendency is to suffer wishful thinking". If it is true that business is nothing more than an amplification of your mind then you better account for your imbalances, limited scope of thinking by applying rigorous scientific method.
I always use Elon Musk as an example because no matter what business he puts his mind to quite simply they work.
Why is someone like Elon able to deliver consistently over a few decades across multiple industries?
This trick also gives you an out when they observe or criticize; it helps you avoid getting defensive or answering too quickly (which will give away this pretense). You can always hedge and redirect by saying, "I'm pretty sure it works like X, but what is natural to you" or "Hmm, I'm not sure, we'll have to ask them-- what do you think is best?"
I'm not necessarily an advocate of "rigor" in applying the scientific method to businesses. I'm more of an advocate of a scientific mindset - a skeptical, curious, and resourceful one.
I don't know much about Elon Musk but would like to know more. Off to Wikipedia :-)
However, there's still a problem. You can facilitate a conversation that uncovers problems people face and the current ways they are addressing the problems. But most people do not accurately predict what they will do in a hypothetical situation.
"Would you buy this product that this other person is developing?" will therefore not yield meaningful insights in most cases.
People are experts on their actual, current situations and challenges. And when confronted with new, real-world choices, they will act. But they do not know how they would act in a hypothetical future situation.