The following is a fictitious example of the type of conversation that could occur at many organizations claiming to use agile development methods.
Joan is the VP of engineering at Trendy Startup, a rapidly-expanding company developing a suite of cloud storage products and services.
Joan's organization uses the popular scrum development process to develop its products. Product managers and owners create user stories, allocate them to iterations and releases, and manage them in backlogs. Quality assurance (QA) engineers write tests for the user stories, and developers implement the user stories. Team members meet daily to review the previous day's accomplishments, agree on what they'll do today, and surface any obstacles they are facing.
Joan is proud of her teams and their adoption of agile methods and practices. She touts the Trendy Startup's development processes to her fellow executives and to prospects and customers.
All members of her product teams have embraced agile methods, but two of her independent-minded engineers, Raquel and Oscar, are frustrated and have begun to question the results. They decide to meet with Joan to discuss their concerns.
Joan: "What's up, Raquel? Oscar?
Raquel: "We're concerned about our development process at Trendy Startup."
Joan: "Raquel, you know we're an agile shop. It's important that we all buy into agile practices, and frankly I don't think there's a place in our company for engineers who can't fit into agile teams."
Raquel: "Oh, I have no problem with agile methods. My concern is that I don't think we're really doing agile."
Joan: "Hmm. I'm not sure what you mean. We've adopted all the major practices, haven't we?"
Raquel: "Sort of. But what we call 'stories' are really just development tasks. We're not defining acceptance criteria for our user stories. And our product roadmap is oriented around features instead of what really matters to users."
Oscar: "Exactly! I'm frustrated because we seem to be losing sight of delivering value to customers. I actually don't care whether we are doing 'theoretically pure' agile, but we've gotten too far in the weeds."
Joan: "Ultimately, we can't deliver value to customers and be successful in the market without dealing with the details; things like font sizes, the positions of buttons and other user interface elements, and even grammar."
Oscar: "Of course. But we need to tie these details back to the larger picture of what the user is trying to accomplish as they are exposed to these details."
Raquel: "Right. That's part of what I mean about not doing agile properly. Many of the stories in our backlog are development tasks to 'move the Cancel button two pixels to the left'. But there is no tie-back to a real user story, which should state what the user wants to do and why she wants to do it."
Oscar: "I would go even further. Even when we do have real user stories, I sometimes lose sight of the big picture."
Raquel: "Good point! We've made some attempts to group user stories under features, but that's not how Mike Cohn says to do it. Cohn says to use epics and decompose them into user stories."
Joan: "But we do use epics. We group our user stories into epics such as 'asynchronous storage' and 'cloud infrastructure'."
Raquel: "But those aren't real epics! A story is a narrative. An epic is a long narrative. Look them up in the dictionary! In agile, a story is a placeholder for a narrative that achieves users' goals, and an epic is a placeholder for a longer, end-to-end narrative that captures and implements the larger goals of the users and buyers."
Joan: "Okay, I have to draw the line here. This organization doesn't care what the real definition of 'epic' is. We're not employing poets or novelists here. And in the final analysis, it doesn't matter whether we're doing what some purists think is agile. We care about producing quality software that will sell in the marketplace."
Oscar: "I couldn't agree with you more on those points, Joan, but I think Raquel has hit on precisely why the team loses the user perspective. Users care about what they're doing and what their goals are, not about 'asynchronous storage'."
Joan: "What would you suggest as an alternative? Can you provide an example of what you consider a true epic, Raquel?"
Raquel: "Here's one: 'As a gadget guy, I want to view and edit my documents from any device connected to the Internet so that I'm not dependent on any particular device for access.'"
Joan: "Okay, that seems to capture succinctly what the user wants to do, and the overarching value of our suite of cloud storage products. Your concerns are starting to become more concrete for me. Raquel, how do you suggest we modify our agile process to address these concerns?"
Raquel pulls up the following diagram from a Mike Cohn presentation on user stories.
Raquel: "Let's make sure we include real epics in our portfolio and product roadmaps. Let's decompose epics into user stories and include the user stories in our backlog. When we need to split a user story to fit into an iteration, our first instinct should be to split it by iteratively strengthening the acceptance criteria, not always by decomposing it along functional lines. All development tasks we include in the backlog should reference the user stories they support. Also, let's follow Mike Cohn's template for epics and user stories to ensure they are understandable to users and are verifiable. We can also group stories according to themes, but let's make sure the themes tie back to prospect problems."
Oscar: "Works for me! I really like the idea of epics standing for lengthy, end-to-end stories. They really tie everything together to show the value to customers and users in such a way that I, as a developer, can see how the details fit into the big picture. As a matter of fact, I've noticed a recurring pattern we could employ."
Oscar proceeds to draw the following diagram on Joan's whiteboard.
Raquel: "Pretty cool, Oscar. Your diagram shows a decomposition of an epic into smaller stories. The epic encompasses what the user ultimately wants to do and the unfortunate - but often necessary - system interactions that administrators must perform behind the scenes to enable the true value for end users. You've divided what we would need to implement into chunks our product owner could put in the backlog and our developers could complete in a single iteration. At the same time, none of the individual user stories by itself delivers what the end user needs. The top-level epic captures all of the stories playing out and satisfying the end user's needs."
Joan: "Now that you've explained them, these ideas make a lot of sense to me. We need to make this a team conversation and decision. I will fully support your starting this conversation with the team and proposing your ideas. I suggest a series of 'brown bag' lunches to discuss our process and possible modifications to it. Let's set the expectation that no decisions will be made at these lunches, but through open sharing of ideas and perspectives, we'll naturally converge on some improvements to our processes."
Joan is the VP of engineering at Trendy Startup, a rapidly-expanding company developing a suite of cloud storage products and services.
Joan's organization uses the popular scrum development process to develop its products. Product managers and owners create user stories, allocate them to iterations and releases, and manage them in backlogs. Quality assurance (QA) engineers write tests for the user stories, and developers implement the user stories. Team members meet daily to review the previous day's accomplishments, agree on what they'll do today, and surface any obstacles they are facing.
Joan is proud of her teams and their adoption of agile methods and practices. She touts the Trendy Startup's development processes to her fellow executives and to prospects and customers.
Joan: "What's up, Raquel? Oscar?
Raquel: "We're concerned about our development process at Trendy Startup."
Joan: "Raquel, you know we're an agile shop. It's important that we all buy into agile practices, and frankly I don't think there's a place in our company for engineers who can't fit into agile teams."
Raquel: "Oh, I have no problem with agile methods. My concern is that I don't think we're really doing agile."
Joan: "Hmm. I'm not sure what you mean. We've adopted all the major practices, haven't we?"
Raquel: "Sort of. But what we call 'stories' are really just development tasks. We're not defining acceptance criteria for our user stories. And our product roadmap is oriented around features instead of what really matters to users."
Oscar: "Exactly! I'm frustrated because we seem to be losing sight of delivering value to customers. I actually don't care whether we are doing 'theoretically pure' agile, but we've gotten too far in the weeds."
Joan: "Ultimately, we can't deliver value to customers and be successful in the market without dealing with the details; things like font sizes, the positions of buttons and other user interface elements, and even grammar."
Oscar: "Of course. But we need to tie these details back to the larger picture of what the user is trying to accomplish as they are exposed to these details."
Raquel: "Right. That's part of what I mean about not doing agile properly. Many of the stories in our backlog are development tasks to 'move the Cancel button two pixels to the left'. But there is no tie-back to a real user story, which should state what the user wants to do and why she wants to do it."
Oscar: "I would go even further. Even when we do have real user stories, I sometimes lose sight of the big picture."
Raquel: "Good point! We've made some attempts to group user stories under features, but that's not how Mike Cohn says to do it. Cohn says to use epics and decompose them into user stories."
Joan: "But we do use epics. We group our user stories into epics such as 'asynchronous storage' and 'cloud infrastructure'."
Raquel: "But those aren't real epics! A story is a narrative. An epic is a long narrative. Look them up in the dictionary! In agile, a story is a placeholder for a narrative that achieves users' goals, and an epic is a placeholder for a longer, end-to-end narrative that captures and implements the larger goals of the users and buyers."
Joan: "Okay, I have to draw the line here. This organization doesn't care what the real definition of 'epic' is. We're not employing poets or novelists here. And in the final analysis, it doesn't matter whether we're doing what some purists think is agile. We care about producing quality software that will sell in the marketplace."
Oscar: "I couldn't agree with you more on those points, Joan, but I think Raquel has hit on precisely why the team loses the user perspective. Users care about what they're doing and what their goals are, not about 'asynchronous storage'."
Joan: "What would you suggest as an alternative? Can you provide an example of what you consider a true epic, Raquel?"
Raquel: "Here's one: 'As a gadget guy, I want to view and edit my documents from any device connected to the Internet so that I'm not dependent on any particular device for access.'"
Joan: "Okay, that seems to capture succinctly what the user wants to do, and the overarching value of our suite of cloud storage products. Your concerns are starting to become more concrete for me. Raquel, how do you suggest we modify our agile process to address these concerns?"
Raquel pulls up the following diagram from a Mike Cohn presentation on user stories.
Raquel: "Let's make sure we include real epics in our portfolio and product roadmaps. Let's decompose epics into user stories and include the user stories in our backlog. When we need to split a user story to fit into an iteration, our first instinct should be to split it by iteratively strengthening the acceptance criteria, not always by decomposing it along functional lines. All development tasks we include in the backlog should reference the user stories they support. Also, let's follow Mike Cohn's template for epics and user stories to ensure they are understandable to users and are verifiable. We can also group stories according to themes, but let's make sure the themes tie back to prospect problems."
Oscar: "Works for me! I really like the idea of epics standing for lengthy, end-to-end stories. They really tie everything together to show the value to customers and users in such a way that I, as a developer, can see how the details fit into the big picture. As a matter of fact, I've noticed a recurring pattern we could employ."
Oscar proceeds to draw the following diagram on Joan's whiteboard.
Raquel: "Pretty cool, Oscar. Your diagram shows a decomposition of an epic into smaller stories. The epic encompasses what the user ultimately wants to do and the unfortunate - but often necessary - system interactions that administrators must perform behind the scenes to enable the true value for end users. You've divided what we would need to implement into chunks our product owner could put in the backlog and our developers could complete in a single iteration. At the same time, none of the individual user stories by itself delivers what the end user needs. The top-level epic captures all of the stories playing out and satisfying the end user's needs."
Joan: "Now that you've explained them, these ideas make a lot of sense to me. We need to make this a team conversation and decision. I will fully support your starting this conversation with the team and proposing your ideas. I suggest a series of 'brown bag' lunches to discuss our process and possible modifications to it. Let's set the expectation that no decisions will be made at these lunches, but through open sharing of ideas and perspectives, we'll naturally converge on some improvements to our processes."
Comments
I am curious as to why Raquel and Oscar are raising these points with the VP of Engineering and coming from the Engineering ranks. Where is the Product Owner and/or Product Manager in these discussions? How come they feel a need to raise these points in a side meeting as opposed to a retrospective. I assume that Trendy Startup is doing them correct? If so, why have the brown bag lunches?
That said, Agile's underlying intention is as a guideline, not a rigid process. Ultimately, the organization should be striving for what works for it not something that conforms to a standard.
The way I have seen (and believe in their use) Epics, Features and Stories laid out is as a series of abstraction layers. Each layer providing more detail. This reduces the level of too-early specificity and decreases the overhead of managing these artifacts in larger systems. Dean Leffingwell Agile Software Requirements
I would actually flip Mike Cohn's diagram on user stories around so that it goes more top down than bottom up. (This is really semantics though)
With this in mind, I would say Joan's initial take on the Agile process was correct. Raquel and Oscar's issue could be alleviated at the story level by better incorporating the user's need there.
You and Oscar agree that conformance to a standard or to definitions is not what should drive the process.
Did you not notice that, while Raquel expressed her concerns by pointing out inconsistencies with standard practices and original definitions, Oscar concentrated on the practical ramifications of the company having deviated from those practices and definitions? Joan was ultimately able to see that both Raquel and Oscar had legitimate concerns and perspectives. Raquel's perspective was theoretical, Oscar's perspective was practical. The theoretical and practical arguments converged and reinforced each other.
I'm glad you mentioned Dean Leffingwell's abstraction layers (epic, feature, and story). Before writing this blog entry, I read how he uses the terms and concepts, and I was disapointed. In his usage, the terms epic, feature, and story refer to "system behaviors". Raquel and Oscar believe that a system-centric approach is fundamentally misguided. So do I.
Oscar felt he and other developers lacked context, because the organization's so-called "stories" were oriented around the system functionality and development tasks instead of around what the user really wants to accomplish. Raquel agreed with Oscar on this point and extended her complaint to the product roadmap, which was oriented around features instead of what "really matters to users".
Did you know Leffingwell gives "video streaming" as an example of epic? That sort of terminology and categorization does not lend itself well to maintaining the customer perspective in planning and building products. Moreover, Kent Beck, who apparently coined the term "user story", believes storytelling sparks more emotional engagement than mere descriptions of system functionality. A more faithful example of an epic would be, “Monitor and Improve Energy Efficiency”. Such an epic would stand for a long narrative in which multiple users interacted with one or more systems to achieve the business value that the name of the epic denotes.
And thought leaders in the marketing and product management community realized decades ago that focusing on "features" has a tendency to:
1. Lead to products that don't solve real problems.
2. Result in marketing messages with limited or no relevance to customers.
3. Stifle innovation.
It's true that some features have tangible and immediately-recognizable benefits to users. Leffingwell gives "spell-checking" as an example of such a feature. But many features do not have such a tangible benefit, at least not ones that are immediately obvious until they are put into the context of something the user is trying to accomplish.
If Trendy Startup is going to ensure its product teams understand and maintain the context and purpose of what they are building, the company needs to adopt the suggestion Raquel proposed and Oscar elaborated. A user story should serve as a placeholder for a narrative expressed from the user point of view. An epic is a longer narrative that encompasses, and a product owner may progressively decompose into, smaller user stories. This progressive decomposition, especially when depicted in the form of a diagram such as the one Oscar drew on Joan's whiteboard, gives the customer and user context at every level.
I'm not sure why Leffingwell chose a rigid, three-tiered framework oriented around system functionality instead of a flexible, customer-oriented framework that stays faithful to the original definitions of agile terms. But it was a major factor in spurring me to write this blog entry.
I've definitely seen this problem play out with team after team, and agree that an epic is, and should be, no more than "a story that happens to be too big to do it one sprint."
I think the issue is that people regularly conflate the concepts of themes and epics.
A theme is a collection of related stories that make sense together - for whatever reason. An epic is a single story that needs to be decomposed in order to implement it piecemeal.
A theme may encapsulate stories for multiple users attempting to realize multiple goals. An epic reflects a single goal (possibly shared by multiple users - leading to an approach to decomposition: do it for user A now (with acceptance criteria 1,2, 5) and for user B later (adding in acceptance criteria 3 and 4, which don't matter to user A).
A theme can be structured in terms of positioning or market message ("better for IT administrators"), or strategic objective ("reach parity with competitor X") or an internal stakeholder-driven objective ("cut operational costs of providing our SaaS by 10% per user").
For me, focussing on delivering values to customers by defining what and why a user whants in Epics and Stories is a greate way Keep on track.
I've used Epics and Stories this way in the last 6 month now and made greate experiences so far. For me as a product manager it makes things a lot clearer in many ways. Thanks for sharing this insight with us.