Tuesday, October 06, 2015

Is Customer Development Pseudoscience?

The “Science” of Lean Startup

Lean startup practitioners embrace the scientific method, seeking the "truth" about what business model and strategy will lead to product success. We do so by:

  • Formulating hypotheses
  • Crafting and running experiments to test them
  • Learning from the experiments
  • Iteratively feeding our learnings back into revised hypotheses

Sounds pretty scientific, at least in spirit, doesn't it? Yet this process actually neglects a key ingredient in the scientists' mode of operation. To identify what’s missing, let’s examine “customer development”.

Customer Development

Steve Blank is one of the pioneers of the lean startup movement. He introduced into the lean startup lexicon the term “customer development”. Customer development consists of sessions and interactions with customers to test hypotheses.

For example, a product manager might interview a prospect, asking if she agrees with the product manager’s hypotheses about the problems she faces or the value the product might bring to the prospect. Or she might “pitch” her product ideas to a prospect, as a sales person might do, ask for payment or a commitment of some sort, and learn from the prospect’s reactions.

Customer development practitioners, according to Blank, start with a relatively specific set of preconceived hypotheses and focus customer sessions almost exclusively on assessing the “validity” and applicability of these hypotheses to customers. As practitioners test the hypotheses, they capture data and may stumble across some insights. But customer development sessions do not include open exploration into customers’ challenges, according to Blank.

In fact, Blank pigeonholes open exploration as part of “design thinking”. He suggests design thinking is, at least in common practice, a lengthy, heavyweight approach to learning, and therefore not appropriate for lean startups. (A comment by my former colleague, Jen van der Meer, aptly explains that this view of design thinking is a caricature.)

Even lean startups should include open exploration in their customer interview and observation sessions, lest they neglect a vast body of important potential insights.

Rummy’s Unknown Unknowns

Former Secretary of State Donald Rumsfeld is famous for, in a press conference, having distinguished among “known knowns”, “known unknowns”, and “unknown unknowns”:

"As we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don't know." - Donald Rumsfeld, February 12, 2002

Known known: Something we know we know.
Known unknown: Something we know we don’t know.
Unknown unknown: Something we don’t know we don’t know.

Learning is the process of converting unknowns (whether known or unknown) to known knowns.

Blank’s Misguided Focus

"Customer Development starts with, 'I have a technology/product, now who do I sell it to?'” - Steve Blank, "Driving Corporate Innovation" blog entry

By focusing customer development on testing preconceived hypotheses, Blank has us concentrate on the known unknowns, or the things we know we don’t know (with sufficient confidence). By definition, a preconceived idea or hypothesis is not, and cannot be, an unknown unknown.

If the body of unknown unknowns were very small relative to that of known unknowns, then we could employ Blank’s approach, and we might eventually converge on a set of hypotheses constituting a successful product strategy.

In reality, though, the typical body of knowledge (and lack thereof) might break down as follows:

In any given domain ripe for innovation, and particularly early in a lean startup process, the body of unknown unknowns may eclipse the body of known unknowns. Moreover, you may think you know the extent of the unknown unknowns, but how would you know you know? Consequently, even if we have a prior set of hypotheses - and even a product based on those hypotheses - we should focus some of our efforts on exploring the vast body of unknown unknowns that exist in a domain or industry.

Focus on the Unknown Unknowns

"Customer interviews are about exploring what you don't know you don't know." - Ash Maurya, Running Lean, page 71

We should focus some of our sessions with prospective or existing customers on exploring the situations and challenges they face, not exclusively on directly testing our preconceived notions of what they may be.

You likely do have a set of preconceived hypotheses, but these hypotheses exist within the context of a domain and set of possible stakeholders. This context helps you narrow the scope of your inquiry: the people you should observe and interview, and which aspects of their lives you should explore.

Much as a scientist conducts field studies, a lean startup practitioner should spend at least some time in the field to explore. Much as a therapist uncovers patients' problems by exploring their past and present lives, a lean startup practitioner should uncover unknown unknowns by exploring prospects' past and present ways of getting jobs done.

By employing an exploratory approach grounded in a prospect’s reality, we’ll not only uncover insights we never anticipated, but we’ll also indirectly test the known unknowns. A prospect’s unprompted description of a painful experience is far more significant than a response to your eager “do you experience this pain” question.


There is no need to debate the semantics of “customer development” and “design thinking”. Steve Blank popularized the term “customer development”, and we can defer to his explanation of its definition and its focus. Regardless of the definitions, however, smart lean startup practitioners should include some open exploration in their approaches to interviewing and observing customers. Failure to do so risks missing important opportunities to create innovative and transformative product experiences.

Monday, August 03, 2015

Why Spreadsheets Suck for Prioritizing

The Goal

As a company executive, you want confidence that your product team (which includes all the people, from all departments, responsible for product success) has a sound basis for deciding which items are on the product roadmap. You also want confidence the team is prioritizing the items in a smart way.

What Should We Prioritize?

The items the team prioritizes could be features, user stories, epics, market problems, themes, or experiments. Melissa Perri makes an excellent case for a "problem roadmap", and, in general, I recommend focusing on the latter types of items. However, the topic of what types of items you should prioritize - and in what situations - is interesting and important but beyond the scope of this blog entry.

A Sad but Familiar Story

If there is significant controversy about priorities, then almost inevitably, a product manager or other member of the team decides to put together The Spreadsheet.

I've done it. Some of the most respected names in product management have done it:
The Spreadsheet is a list of all the ideas for the product roadmap or "backlog", along with columns for every factor imaginable that might affect the priority of the items.

Columns in The Spreadsheet might include:
  • Man-hours to implement
  • Tool and other expenses required to implement
  • Potential revenue
  • Number of customer requests
  • Number of lost deals due to absence of feature
  • Alignment with quarterly or annual corporate strategic priorities
Someone creates a "magic formula" in the final column of the sheet that combines and weighs the various factors in the other columns and computes a "score". Then the product manager or team and you all get in a room and fill in the sheet. The group proceeds to debate the finer points of, for example, assigning a "5" or a "6" rating to the "potential revenue" column for a particular feature idea.

Halfway through the multi-hour exercise, you and the team are fatigued and decide to pick it up another day. Whether or not this follow-up meeting ever happens, the team ignores much of the priorities that result from the effort, and few if any updates ever occur to The Spreadsheet.

Why "The Spreadsheet" Sucks

The Spreadsheet approach to prioritization has at least three fatal flaws:
  1. Organizational dysfunction. It attempts to address organizational dysfunction with formulas and analysis that ignore human factors.
  2. Product strategy void. It indicates team members lack a shared understanding of the product strategy.
  3. Distraction from unique value proposition. It distracts from your product's value proposition or from the target audience that derives the value from your product.
Let's examine each of these flaws in more detail.

Organizational Dysfunction

Step back and consider why one or more people on the team thought The Spreadsheet was a good idea. In all likelihood, it was due to difficulty determining or agreeing on priorities.

At some point, you may have asked your product manager, "Why aren't we working on feature Y instead of feature X? What data do you have to back up your decision?" Or members of the sales team bypass the product manager, talk directly to developers, and tell them how the next big deal will happen if only they would implement a killer feature. Or multiple product managers are competing for development resources and employ passive aggressive tactics to divert those resources to work on their products.

By now, you recognize the recurring theme: dysfunctional interactions among team members. The hope is that The Spreadsheet causes everyone to set aside their distrust and passive aggression and become "objective". If we just focus on the numbers and let the "magic formula" compute the score, all these dysfunctional interactions will go away, and we'll get on the same page.

Once you recognize the context (dysfunctional team interactions) and the proposed solution (objective analysis of priorities), it becomes obvious to anyone with a modicum of human sensibility that the proposed solution misses the mark.

Product Strategy Void

Why does this distrust and passive aggression exist in the first place? Or if no such dysfunctional behavior is occurring, why is it nevertheless so hard to get on the same page? It could be due to personalities and culture, or it could be that the team simply doesn't have a shared understanding of the product strategy.

One model of product management is that the product manager or product owner (The All-Knowing One or The Decider, as Teresa Torres might say in jest) knows best, prioritizes a backlog, and doles out the work to designers and developers. "I've prioritized it for you; just design and implement it."

This approach works fine when the product manager or owner is, in fact, all knowing, has the time to meticulously specify and prioritize every user story in the backlog, and when the designers and developers are machines that do exactly what they're told. Good luck with that.

What happens in the real world? The All-Knowing One is swamped, she has no time to talk to prospects and customers, and designers and developers keep pestering her with questions. When the product manager is wrong or has no sound basis for her answers, she loses credibility.

May I suggest another approach?

Teams are most productive when they share a strategic vision and each team member can "fill in the blanks". What if the product manager facilitated a shared understanding of the product strategy? Then each member of the team would feel confident making decisions without going back to The All-Knowing One. And the prioritizing that team members do on a daily, or even hourly, basis would flow naturally out of that strategy.

In the absence of a shared understanding of the product strategy, you, your product manager, and team members aren't likely to agree on priorities. The Spreadsheet isn't a strategy and is no substitute for a strategy. It won't fill this fundamental void.

Distraction from the Unique Value Proposition

Let's say you do have a product strategy and a shared team understanding of what it is. A fundamental part of your product strategy is the unique value proposition.

The unique value proposition is the value you envision your product uniquely delivering to a target market.

When the team chooses the product's unique value proposition, it selects a singular, overarching theme that implicitly embodies the set of market problems it plans to solve with the product. It has largely selected the high-level priorities and deliberately set aside problems and features that don't support the overarching theme.

For more information on unique value propositions and how to position your product relative to the competition, see this entry on competitive mindshare maps.

Imagine sales and customers bombard the team with requests for a "killer feature" that has little or nothing to do with the unique value proposition. How does the feature fare in The Spreadsheet?

Well, it scores well in the "number of customer requests" category. Perhaps it has great "revenue potential" as well, so bump up the score in that column, too. Sales claims it loses a lot of deals due to the lack of the feature. It looks like this feature might score very well indeed on The Spreadsheet.

But evaluating the feature through the lens of the unique value proposition might lead to a very different assessment of its priority. In fact, the feature might undermine, or be unrelated to, the unique value proposition.

Now, given the feedback you've gotten from customers and sales, you might want to consider whether to "pivot" and change the unique value proposition, and thereby the product strategy, for the product. Alternatively, you might consider spinning off another product or company with a fresh and separate set of product strategy hypotheses based on the new information.

But if you believe you already have a product strategy worth pursuing and testing, The Spreadsheet has led you astray. If you prioritize features without regard for the unique value proposition, you end up with a fractured product that distracts from - rather than delivers - its promised value. It may deliver bits and pieces of value, but those pieces have little relation to each other. They don't coalesce around a central theme.

While the "potential revenue" rating might nonetheless seem to justify the high priority, it neglects the power of brand: the long-term impact on revenue of a cohesive product that delivers a clear value proposition.

How to Prioritize

At this point, you're wondering how your team should prioritize, if not using The Spreadsheet.

I propose three primary questions to guide prioritization:
  1. To what extent does this item (feature, user story, epic, market problem, theme, or experiment) support the unique value proposition?
  2. What is the level of effort required to implement this item?
  3. What will implementing this item enable us to learn?
(Heck, I might throw in a fourth one: how fun will it be to work on this item? It matters.)

The answers to these questions aren’t numbers team members plug into a formula. They form the basis for thought processes and conversations. Consider them holistically and always in the context of the unique value proposition. As Teresa Torres cautions, "If you use time-to-build to prioritize what to build next, you’ll end up with a product full of easy solutions." Teresa rightly urges us to emphasize impact on a goal. I contend the goal is delivering the unique value proposition, as I wrote in 2013.

Accordingly, your product manager should lead the process of determining the product strategy (including the unique value proposition) and foster a shared understanding of it, and the reasons for it, among the entire team (across all departments). The product manager may also lead the process of conceiving experiments to continually test the riskier assumptions behind the product strategy hypotheses.

This high-level strategic baseline sets the stage for every member of the team to make common sense day-to-day prioritization decisions without resorting to numbers and formulas and without going back to The All-Knowing One for guidance. And when the product manager or team prioritizes one item over another, it will flow naturally out of the shared understanding of the product strategy. Steve Johnson calls it "empowering judgment with context".

Healthy conversations about how my three considerations apply to an item should and will occur. But rarely will you need to ask for the "data" that justifies a prioritization decision. The Spreadsheet approach won't appeal to anyone, except perhaps to the most hopelessly analytical members of the team. God bless 'em.

Sunday, January 04, 2015

What Is Design Thinking?

The Context

Over the years, various product management and development methods have come into vogue, most notably agile and lean startup methods. Agile methods addressed requirements, architecture, and development risks using frequent iterations. Lean startup methods take it a step further, iterating on the entire product strategy, including the unique value proposition, the target market, revenue and cost structure, and marketing and sales channels.

Recently, you may have heard more and more chatter about "design thinking". I certainly have. What is it, and how does it differ from agile and lean startup methods? How can companies leverage it to innovate and develop better products?

The Definition

If you do some web searches, you'll notice a lot of vague references to design thinking and a diversity of views about what it is. After exploring some definitions, I'll borrow from various descriptions and accounts of design thinking to form what I believe is a coherent explanation of what it is.

Let's see how Tim Brown, CEO of IDEO, defined "design thinking" in his seminal article in Harvard Business Review:
"[Design thinking is] a methodology that imbues the full spectrum of innovation activities with a human-centered design ethos."
Clear as mud, eh?

My former colleague, Davide Casali, defines it as follows:

This definition is concise and nicely aligns with the plain meanings of the words. It leads us to the key question: what are the characterizing activities of design thinkers? Indeed, Wikipedia states:

Hmm. I'm not sure about the "design-specific" part of this definition, but let's look at the activities that characterize design thinking.

The Activities

Note: these activities characterize design thinking but do not necessarily come in any particular sequence or imply a particular "process".
  • Explore the problem domain. Interview prospective customers and users. Observe and experience the problem and its context first hand.
  • Define the problem you are trying to solve. Half of the battle in designing a great solution is to clearly define the problem you're solving.
  • Ideate to generate perspectives on the problem and ideas for solving it. Set aside analytical thinking and engage your right brain.
  • Hypothesize about the problem, solution, and business model. What would have to be true for the problem to resonate with prospects and for the solution to solve the problem? How would you know that your hypotheses and assumptions are false?
  • Experiment to test your hypotheses and assumptions. Employ prototypes and confront prospects with real and concrete choices.
  • Learn from all your other activities, especially from exploration and experiments. Design thinking isn't (merely) about "creativity" and having a sense for the "right" solution. It's about learning and discovery.
Key Concepts

Design thinkers leverage key concepts that differentiate the solutions that emerge from their activities.
  • Holism. In design thinking, the product is a set of experiences, not of features, that provide value. The nature and value of the experiences depends on a larger ecosystem, such as the one Apple created with iTunes.
  • Iteration. You almost never get it right the first time, and you shouldn't spend a lot of time trying. While design thinkers don't necessarily employ a particular process, they iterate in their approaches to understanding the problem, the customer, and the solution.
Final Thoughts

Some product managers, agile teams, and lean startup practitioners engage in some of the same activities, and apply some of the same concepts, as design thinkers. In particular, iteration has long been a core part of agile processes. Lean startup practices are iterative and also incorporate hypothesizing, experimenting, and learning in a build-measure-learn loop. The best product managers learn about markets by interviewing and observing prospective customers and users.

But agile and lean startup methods differ from design thinking in several ways:
  1. Though certain activities and concepts characterize design thinking, it's not a "process".
  2. While agile methods are iterative and include some hypothesizing, experimenting, and learning, they don't necessarily include other design thinking activities or a holistic view of the product.
  3. Open-ended exploration is not part of lean startup methods, though they arguably incorporate all other design thinking activities. (I'll address this controversial topic in a future blog entry. UPDATE: Read how the "customer development" concept in lean startup methods, at least according to Steve Blank, does NOT include open-ended exploration.)
  4. You can employ lean startup methods without a holistic view of what constitutes a "product".
Many product teams suffer from tunnel vision, which stifles innovation. The tunnel vision stems from paying insufficient attention to raw exploration and obsessing over "validating" their preconceived ideas. And most of them treat products as collections of features instead of holistic sets of experiences. Product managers can up their game by leveraging all aspects of design thinking.

UPDATE: I covered design thinking at ProductCamp Austin 14. The title of my session was "WTF Is Design Thinking". You can get a link to the slides by tweeting this blog entry. Send this tweet:
What Is #designthinking? http://cauv.in/wtfdt
Thanks to Davide Casali and Shardul Mehta for reviewing and providing feedback on initial drafts of this blog entry.