Skip to main content

Stop Validating and Start Falsifying

The product management and startup worlds are buzzing about the importance of "validation". In this entry, I'll explain how this idea originated and why it's leading organizations astray.

Why Validate?

In lean startup circles, you constantly hear about "validated learning" and "validating" product ideas:


The assumption is that you have a great product idea and seek validation from customers before expending vast resources to build and bring it to market.

Indeed, it makes sense to transcend conventional approaches to making product decisions. Intuition, sales anecdotes, feature requests from customers, backward industry thinking, and spreadsheets don't form the basis for sound product decisions. Incorporating lean startup concepts, and a more scientific approach to learning markets, is undoubtedly a sounder approach.

Moreover, in larger organizations, sometimes further in the product life-cycle, everyone seems to have an opinion about such aspects of the business model as:
  1. The most pervasive and urgent market problems the product should solve.
  2. Whether the solution truly addresses the market problems.
  3. What price customers would be willing to pay.
  4. What tactics will most effectively drive prospects to buy.
Presumably, the "validated learning" approach of lean startup enables the organization to identify which opinions are factual and not mere speculation.

Validation in Practice

But let's consider what our naive lean startup practitioner (we'll call her "Cameron") does in practice.


Present the idea





Cameron puts together a slide deck, minimum viable product (MVP), or demo, presents her idea to one or more prospects, and eagerly awaits feedback. The prospects say, "I like it!" or "I would buy it!" Cameron feels warm inside, as the prospects have "validated" her and her idea.



Ask about the "pain point"





Since solving an urgent and pervasive problem or "pain point" is usually a prerequisite for product success, Cameron visits with prospects and asks them if they experience one of the problems she envisions her product would solve. "Yes," reply the prospects. Feeling "validated", Cameron does an internal high-five with herself.



Name a price





Cameron asks prospects what they would pay for a solution with the features she envisions for her product. Prospects "name their price". Or, in equally naive fashion, Cameron herself names a price and asks, "Would you pay X for the product?" Prospects reply affirmatively. Cameron can hardly contain her giddyness as her pricing assumptions are "validated".

Cameron feels extra special for having "gotten out of the building" to visit customers.

What's Naive about Cameron's Approach?

The results of all of Cameron's efforts are practically worthless, aside from the emotional affirmation she may feel. When you visit prospects, you don't get reliable information by posing hypothetical questions. When you seek validation from someone, you will tend to get it. Cameron unwittingly designed her interactions with prospects in a manner likely to confirm her preconceived ideas, and her interpretations of the results were naive. Let's call it "validation bias", an insidious psychological disorder that has infected the lean startup community.

Note: had Cameron conducted a survey, though seductive because it would have yielded quantitative data, the results would still have been suspect to the extent the questions were hypothetical and failed to confront prospects with real choices and commitments.

What's the Alternative?

If you want to be scientific in your approach to product decisions, you craft experiments and make falsifiable predictions, with the intention of testing, not "validating", the hypotheses and underlying assumptions.

Philosopher Karl Popper is famous for having championed falsificationism, a set of principles that shifted the emphasis from verifying scientific beliefs to ensuring they are possible to falsify via experiments:

According to Wikipedia:
"A statement is called falsifiable if it is possible to conceive an observation or an argument which proves the statement in question to be false."
In fact, Popper argued that a statement isn't scientifically factual at all if it isn't falsifiable.

What Cameron Could Do Differently

Instead of asking hypothetical questions, Cameron could ask the prospects what they actually do (and have done) instead of what they would do. She could probe into their current and past experiences and note whether the supposed problems manifested themselves in those experiences. She could sit with prospects in their native environments and observe (ethnography), first hand, their situations and challenges. Cameron could test her value proposition and pricing hypotheses by quantifying the costs that problems and their workarounds impose on prospects. She could ask for an actual commitment to pay for the product once it's released.

Cameron could also work with her team to craft "digital" experiments and predict how those experiments will turn out. For example, her team could author "how-to" guides, downloadable on landing pages, that help prospects solve the problems she assumes they face. The team could predict how many people will visit those landing pages and how many people will proceed to download the guides. (To make prospects aware of the how-to guides, the team could run Facebook ads or Google AdWords to drive people to those landing pages and see if the predicted number of page visits and downloads materializes.)

The possibilities are endless, but they require a mindset that emphasizes falsifying product ideas and business model hypotheses, not "validating" them.

Comments

Unknown said…
Roger, great post! It's spot on and a distinction people often overlook.
brian piercy said…
Roger, I also like this post - but I'm a bit slow. It seems like most customers would provide "falsified" feedback on one or more features/core concepts. (After all, customers come in all shapes & sizes.) What am I missing?
Roger L. Cauvin said…
Brian, thanks for the question.

To conduct interviews and experiments that help us truly test our business model, we have to ensure we're not biasing the responses.

When a prospect is sitting with you, and you ask her if she likes your idea, it's your idea. If she says she doesn't like it, she'll in many cases be uncomfortable trashing your idea.

If you ask if the your assumptions regarding the problems she faces are valid, she'll similarly be uncomfortable telling you you're wrong.

Moreover, prospects aren't experts on product ideas and business models. They're experts on what the have done, and what they currently do.

So you need to craft your questions in a way that doesn't bias responses, and you have to focus on actual experiences, not hypotheticals.

I wrote about many of these issues in my entry on the top mistakes product managers make when conducting prospect interviews.
Unknown said…
Like it, Roger. Actually, We are conducting a Survey from Our perceived Customer Segment, and this post will definitely help to redesign it to seek real inputs. Thank You.
sachin kundu said…
As always asking the right question will only get you the right answer. That is why Lean Startup is easy to understand and difficult to implement.

Anyways LS clearly says experiments must be falsifiable.

For example problem interview must ask provide an answer to questions like

Do you have the specific problem I am trying to solve. Answer is either yes or no. An objective measurable response.

How do you solve the problem currently, confirms the alternatives. Again very precise.

And so on. Its true to making experiments which are falsifiable is heart breaking perhaps for entrepreneurs but then not forming is their fault not of the methodology.

Roger L. Cauvin said…
Thanks, Sachin. We agree that lean startup methods aren't to blame. It's the implementation of those methods that can lead organizations astray.

However, neither of the sample questions you posed is likely to yield reliable information. While it's possible that the prospect will answer "no", they may be hesitant to let you down with a negative answer.

Instead, I recommend asking open-ended questions about what prospects actually do, not posing your assumptions and eagerly (and obviously) awaiting a positive response.
Roger L. Cauvin said…
Here's a great YouTube video illustrating how "confirmation bias" is part of human nature, and we tend to craft our experiments to verify, rather than falsify, our hypotheses: http://www.youtube.com/watch?v=vKA4w2O61Xo.
Unknown said…
Great post Roger, this is probably painful reading for many would-be entrepreneurs that all too often shift back into their emotive way of feeling rather than applying critical thinking. I'm always reminded of the stats, over 90% of SUs will fail to return initial capital, emotions are the reaons why.

After reading the LeanSU manual front to back it's clear to me how many entrepreneurs haven't bothered to read the materials. Those that have, how many take it as a cast iron blueprint to success whilst jumping on the hype bandwagon after attending a few seminars/workshop surrounded by positive high fives. Reminds me of the Tony Robbins self help crew.

I hear these terms like lean SU, validated learning, actionable metrics knowing that most have very little deep scientific understanding about implementation or why it's so vital.

The problem with many entrepreneurs is themselves which they never factor in their equation, hence the absolute need for rigorous scientific analysis.

As Elon Musk explains "natural human tendency is to suffer wishful thinking". If it is true that business is nothing more than an amplification of your mind then you better account for your imbalances, limited scope of thinking by applying rigorous scientific method.

I always use Elon Musk as an example because no matter what business he puts his mind to quite simply they work.

Why is someone like Elon able to deliver consistently over a few decades across multiple industries?
jaime said…
here's a trick. When conversing with someone as YOUR idea. Pretend it's from some anonymous third party committee who's neither an idiot nor a genius, keep it neutral. Then, the person you're speaking with will not worry about being polite to you (when seeking validation) or avoiding insulting you (when seeking falsification). As a facilitator of the conversation, always maintain that the designer is an anonymous committee, no individual; thus all those "nice" social behaviors will hopefully be less potent. Also frame the questions such that the "nice" participant is "speaking for the masses of users"; then they might *take advantage* of their empathy and speak more truthfully out of concern for the masses over just the designer.

This trick also gives you an out when they observe or criticize; it helps you avoid getting defensive or answering too quickly (which will give away this pretense). You can always hedge and redirect by saying, "I'm pretty sure it works like X, but what is natural to you" or "Hmm, I'm not sure, we'll have to ask them-- what do you think is best?"
Roger L. Cauvin said…
Daniel, a very belated thanks for your comment and kind words.

I'm not necessarily an advocate of "rigor" in applying the scientific method to businesses. I'm more of an advocate of a scientific mindset - a skeptical, curious, and resourceful one.

I don't know much about Elon Musk but would like to know more. Off to Wikipedia :-)
Roger L. Cauvin said…
Jaime, thanks for sharing your "trick". It does help to get more objective feedback on an idea if the audience doesn't think you'll take it personally.

However, there's still a problem. You can facilitate a conversation that uncovers problems people face and the current ways they are addressing the problems. But most people do not accurately predict what they will do in a hypothetical situation.

"Would you buy this product that this other person is developing?" will therefore not yield meaningful insights in most cases.

People are experts on their actual, current situations and challenges. And when confronted with new, real-world choices, they will act. But they do not know how they would act in a hypothetical future situation.
Tristan said…
Excellent post! Karl Popper is one of my favorites.
Great! Scientific and philosophical.... That the way to be practical and sensible.

Sorry .......... That IS the way.... "is" was missing.
Zakir Jaafar said…
Great article and still valid. It will stand the test of time Falsifying and not Validating. I have done my fair share of mistakes over the years in Validating. I understand what Roger is saying. Ther's so main bais coming out from Validating.

Popular posts from this blog

Why Spreadsheets Suck for Prioritizing

The Goal As a company executive, you want confidence that your product team (which includes all the people, from all departments, responsible for product success) has a sound basis for deciding which items are on the product roadmap. You also want confidence the team is prioritizing the items in a smart way. What Should We Prioritize? The items the team prioritizes could be features, user stories, epics, market problems, themes, or experiments. Melissa Perri  makes an excellent case for a " problem roadmap ", and, in general, I recommend focusing on the latter types of items. However, the topic of what types of items you should prioritize - and in what situations - is interesting and important but beyond the scope of this blog entry. A Sad but Familiar Story If there is significant controversy about priorities, then almost inevitably, a product manager or other member of the team decides to put together The Spreadsheet. I've done it. Some of the mos

Is Customer Development Pseudoscience?

The “Science” of Lean Startup Lean startup practitioners embrace the scientific method, seeking the "truth" about what business model and strategy will lead to product success. We do so by: Formulating hypotheses Crafting and running experiments to test them Learning from the experiments Iteratively feeding our learnings back into revised hypotheses Sounds pretty scientific, at least in spirit, doesn't it? Yet this process actually neglects a key ingredient in the scientists' mode of operation. To identify what’s missing, let’s examine “customer development”. Customer Development Steve Blank is one of the pioneers of the lean startup movement. He introduced into the lean startup lexicon the term “customer development”. Customer development consists of sessions and interactions with customers to test hypotheses. For example, a product manager might interview a prospect, asking if she agrees with the product manager’s hypotheses about the problem

Interaction Design: the Neglected Skill

Your product development organization has a big, gaping hole in it. (Be prepared to feel defensive as you continue reading.) One of the most important roles in product development is the role of interaction designer. An interaction designer designs how the users will interact with the product and conceptualize the tasks they perform. He decides whether, for example, the user interface will be command driven, object oriented (clicking on objects then specifying what to do with them), or wizard based. The interaction designer decides the individual steps in the use cases. Every company has one or more people that play the interaction designer role. Usually, those people have little or no expertise in interaction design. Sadly, they typically don't even realize how unqualified they are. Let's see who typically plays the role at companies. Engineer . An engineer is an expert on building what is designed. Yes, an engineer may know how to design the internal structure of the hardware