Agile Testing: A Practical Guide for Testers and Agile Teams

Chapter 15 Tester Activities in Release or Theme Planning

Agile development teams complete stories and deliver production-ready software in every iteration but plan the big picture or a larger chunk of functionality in advance. A theme, epic, or project may encompass several iterations. In this chapter, we look at what testers do when their team takes time to plan their release. We also consider ways to track whether our development is proceeding as anticipated, or if course corrections are needed.

The Purpose of Release Planning

One reason software teams try agile development is because they know long-range plans don’t work. Most business environments are volatile, and priorities change every week, or even every day. Agile development is supposed to avoid “big design up front.” Most of us have experienced making plans that turned out to be a waste of effort. But we have to have some understanding of what our customer is looking for and how we might deliver it in order to get off to a good start. Fortunately, an agile approach can make planning a useful way to give us a head start on knowing how we will deliver the product.

Agile Planning Applied

Janet’s sister, Carol Vaage, teaches first grade when she isn’t directing conferences. She relates her first experience with using agile practices to organize a conference:

My table is loaded with binders and to-do lists, and a feeling of being overwhelmed freezes me into inaction. I am Conference Director and the task right now seems onerous. When my sister offers to help me, I agree, because I am desperate to get this planning under control. I welcome Janet to my clutter, show her my pages of hand-written lists of things that need to get done, explain the huge tasks waiting for my attention, and share how my committee works.

Janet showed me in simple language how to separate each task onto a sticky note and use color coordination for different responsibilities and different individuals. She explained about the columns of “To Do,” “In Progress,” “To Review,” and “Done.” I had never heard of the word iteration before but fully understood about a timeline. She recommended two-week blocks of time, but I chose one-week iterations. We set up a wall for my planning board, and Janet left me to pull it together and to add the tasks needed.

In the six days since Janet has been here, ten tasks have been moved from the To-Do column to In-Progress. Three tasks are Done, and specific time-related tasks have been blocked by the correct time period. The most positive thing is that as I add more tasks in the To-Do column, I am not feeling overwhelmed. I understand that all I need to do is initiate the steps to start it, and then the job becomes easier. The feeling of chaos is gone; I see progress and understand that there is still much work to be done. The timeline is clear, the tasks are discrete and concrete. And the most difficult task of all, finding a way to coordinate the video conference for our keynote speaker has been tackled. This system works!

Agile planning and tracking practices are useful for more than software development. A little time carefully invested, and simple tools used in organizing and planning the testing activities and resources for a release, will help the team deliver high-quality software.

XP teams may take a day every few months for release planning. Other agile teams do advance planning when getting ready to start on a theme, epic, or major feature, which we think of as a related group of stories. They work to understand the theme or release at a high level. What is the customer’s vision of what we should be delivering? What’s the purpose of the release? What’s the big picture? What value will it deliver to the business, to the customers? What other teams or projects are involved and require coordination? When will UAT take place? When will code be released to staging, to production? What metrics do we need to know if we’re on track? These general questions are addressed in release planning.

Some teams don’t spend much time doing release planning activities. Priorities change quickly, even within a particular theme of features. Nobody wants to do too much work up front that ends up being wasted. Some teams just look at the first couple of stories to make sure they can get a running start. At the very least, teams want to know enough to get their system architecture pointed in the right direction and get started on the first few stories.

These planning meetings aren’t intended to plan every iteration of the release in detail. And we know we can’t predict exactly how many stories we can complete each iteration. However, we do have an idea of our average velocity, so we can get a general idea of the possible scope of the release. The team talks about the features and stories, trying to get a 20,000-foot view of what can go into the release and how many iterations it might take to complete. Both of us like Mike Cohn’s approach to release planning in his book Agile Estimating and Planning [2005]. Stories that the business wants to include are sized relative to each other, and then features are prioritized according to the value they deliver. The team may identify “thin slices” through the features to determine what stories absolutely have to be done, what’s in scope, what “nice-to-haves” could be put off until later. They look at dependencies between stories, relative risk, and other factors that determine the order in which features should be coded. The order in which stories are coded is as important, or sometimes more important, than the size of the stories. Teams want to deliver value the first iteration of the release.

Release planning is a chance for the developers and customers to consider the impact of the planned features on the larger system, clarify assumptions, and look at dependencies that might affect what stories are done first. They may think about testing at a high level and whether new resources such as test environments and software will be needed.

Let’s follow our agile tester through release planning activities and see how she contributes value through her unique perspective and focus.

Sizing

Agile teams estimate the relative size of each story. Some teams size as they go, delaying the estimation until the iteration where they’ll actually complete the story. Others have meetings to estimate stories even in advance of release planning. Some developer and customer teams sit together to write and estimate the size of stories all at one time. The goal of sizing is for the programmers to give the business an idea of the cost of each story and to help them prioritize and plan the first few iterations. High-functioning teams who’ve worked together for years may take a less formal approach. For new agile teams, learning to size stories takes a lot of practice and experience. It’s not important to get each story sized correctly but to be close enough to give customers some idea of how big the stories are so they can prioritize with better information. Over time, variations on individual story sizing will average out, and we find that a theme or related group of stories takes about the amount of time expected.

How to Size Stories

As far as how to calculate story size, different teams use different techniques, but again, we like Mike Cohn’s approach to determining story size. We size in story points, ideal days, or simply “small, medium, large.” The relative size of each story to others is the important factor. For example, adding an input field to an existing user interface is obviously much smaller than developing a brand new screen from scratch.

If the business knows the average velocity (the number of story points the team completes each iteration) and has the initial size estimates of each story it wants to get done, it has an idea of how long it might take to implement a given theme. As with any other development methodology, there are no guarantees, because estimates are just that. Still, the business can plan well enough to conduct its usual activities.

Our teams use planning poker (explained in Mike Cohn’s book Agile Estimating and Planning) to estimate story size. In planning poker, each team member has a deck of cards. Each card has a number of points on it. The process begins with the customer or product owner reading a story and explaining its purpose and the value it will deliver. He might list a few conditions of satisfaction or high-level test cases. After a brief discussion, team members each hold up a point card that represents how “big” they think the story is from their perspective. They discuss any big differences in point value and estimate again until they reach consensus. Figure 15-1 shows team members talking about the point values they each just displayed. This needs to be a quick process—long discussions about details don’t result in more accurate size estimates.

Figure 15-1 Planning poker. Used with permission of Mike Thomas. Copyright 2008.

Some teams figure the relative sizes of stories by how many people are needed to complete a given story in a set amount of time. Others estimate how many ideal days one person would need to finish it. Use a measurement that makes sense to all team members and one that provides consistency among estimates.

The Tester’s Role in Sizing Stories

One of our favorite sayings is, “No story is done until it’s tested.” However, we’ve run across teams where testing wasn’t included in estimates of story size. In some cases, testing a piece of functionality might take longer than coding it.

In our experience, testers usually have a different viewpoint than other team members. They often have a broad understanding of the domain and can quickly identify “ripple effects” that one story might have on the rest of the system. They also tend to think of activities not directly related to development that might need to be done, such as training users on a new or changed interface.

Lisa’s Story

What does a tester do during the story sizing process? I think quickly about the story from several viewpoints. What business problem is the story solving, or what business value does it deliver? If this isn’t clear, I ask the product owner questions. How will the end user actually use the feature? If it’s still not clear, I ask the product owner for a quick example. I might ask, “What’s the worst thing that could go wrong with it?” This negative approach helps gauge the story’s risk. What testing considerations might affect the story’s size? If test data will be hard to obtain or the story involves a third party, testing might take longer than coding. I try to quickly flush out any hidden assumptions. Are there dependencies or special security risks? Will this part of the application need to handle a big load?

Many stories aren’t big enough to warrant that much thought. Usually, we don’t need much detail to get an idea of relative size. However, your team can really get burned if a story is underestimated by a factor of five or ten. We once gave a relatively small estimate to a story that ended up being at least ten times the size. These are the disasters we want to avoid by asking good questions.

—Lisa

Testers need to be part of the sizing process. Some teams think that only programmers should participate, but when testers are active participants, they can help to get a much more accurate story sizing, which is in the best interests of the whole team.

An Example of Sizing Stories

Let’s imagine we have the story in Figure 15-2 to size up.

Figure 15-2 Story to delete items

After the product owner reads the story, the following discussion ensues:

Product Owner: “We just want some easy way for users to delete items, but we don’t have a specific implementation in mind.”

Tester: “Should they be able to delete several items at once?”

Product Owner: “Oh, yes, just make it as easy as possible.”

Tester: “What if they accidentally delete an item they wanted to buy?”

Product Owner: “Is there some way the deleted items can be saved for later retrieval?”

Programmer: “Sure, but you should write a new story for that. For now, we should start with the basic delete functionality.”

Tester: “Last release we implemented a wish list feature. Do you want users to be able to move items from their shopping basket to their wish list? That would be a new story also.”

Product Owner: “Yes, those are two more stories we want to do, for sure. I’ll write those down, we can size them also. But we could definitely put them off until the next release, if we have to.”

Tester: “What’s the worst thing that could happen with this feature?”

Product Owner: “If they can’t figure out how to delete, they might just abandon their whole shopping basket. It has to be really easy and obvious.”

The ScrumMaster calls for an estimate. The team understands they’re sizing only the basic story for deleting items, not for doing something else with the deleted items. They quickly agree on a point value.

Let’s look at another story. (See Figure 15-3.)

Figure 15-3 Story on shipping speed

Tester: “What are the shipping speeds the user can choose?”

Product Owner: “Standard 5-day, 2-day, and next-day.”

Programmer: “We should probably start by only offering one speed, and calculating that cost. Then we can easily implement the other two speeds.”

Product Owner: “It’s fine to break it up like that.”

Tester: “Will we use BigExpressShipping’s API to calculate cost based on weight and destination?”

Programmer: “That would be the easiest.”

The team holds up their point cards. The tester and one of the programmers hold up an 8; the other developers hold up a 5.

ScrumMaster: “Why did you two choose 8?”

Tester: “We’ve never used BigExpressShipping’s cost API before, and I’m not sure how that will impact our testing. We have to find out how to access their system for testing.”

Other Programmer with 8: “I agree, I think the testing effort is more intense than the coding effort for this story.”

The team agrees to size the story as eight points.

This sizing process may occur before the planning meeting, and if the stories were sized or estimated a long time ago, the team might want to make sure they feel comfortable with the story sizes. Teams may have changed or may be more experienced. Either of those factors can make a team change the estimates.

There are many times when a story will have a large testing component, and the coding effort is small. At other times, the reverse will be true. It’s important to consider all perspectives.

Lisa’s Story

Our team grew to dread story sizing meetings, because we got into overly long discussions about details, and the meetings always lasted long past the scheduled time. Since then, our ScrumMaster has found ways to keep us on track. She uses an egg timer to time discussions, and stops them each time the sand runs out to see if we think we really need more time to ask questions. Our product owner has also learned what information we need for estimating and usually has what we need. We also learned to only work on stories that were likely to come up in the next few iterations.

With all of our meetings, little traditions have grown to make the meetings more fun. Someone always brings treats to iteration planning meetings. In stand-up meetings, we pass around a combination penlight and laser pointer, so each of us holds it as we report on what we’re working on. We always end story sizing meetings with a competition to see who can throw his or her deck of planning poker cards into the small plastic tub where they live. Figure 15-4 shows this goofy but fun meeting-ending activity. Always remember the agile value of enjoyment and have some fun with your meetings.

—Lisa

Figure 15-4 A meeting-ending tradition. Used with permission of Mike Thomas. Copyright 2008.

Prioritizing

The purpose of the release planning meeting is also to get an idea of what stories the team will try to finish by the release date. The customers prioritize the stories, but there may be dependencies, so it makes sense to do certain stories first, even if they aren’t the highest priority. It is important that the team understands the possibility that not all of the stories will get completed by the release date. One of the basic premises of agile is to deliver working software, so it is important to have the highest-value stories completed first so that the software we do deliver meets the customer’s needs.

Why We Prioritize Stories

Everyone’s goal is to deliver real value in each iteration. Testers can help the team pick out the core functionality that has to work. In Chapter 8, we explained the “thin slice” or “steel thread” concept, identifying one path through the functionality to code and test first, and adding more features after the first critical path works. This concept applies at the release level, too. The order of the stories is critical. Lisa’s team will sometimes break up a story and pull out a core part of a feature to do in the first iteration.

Some teams that don’t do full-blown release planning do take time to look at the stories and decide which two or three should be first. That way, they deliver business value in the very first iteration of the release.

Let’s look at an example.

If our theme is providing the ability for an online shopper to choose shipping options and then calculate the shipping cost based on weight, shipping speed, and destination, it may be a good idea to complete simple stories or even subsets of stories so that the checkout process can proceed end-to-end. Start by only allowing standard 5-day shipping, items less than 10 pounds, and destinations in the continental United States. When the user can get the shipping cost for that scenario and check out, the team can decide the next priorities. They may include heavyweight items, faster shipping speeds, shipping to Hawaii and Alaska, and shipping to Canada and Mexico.

By providing this thin slice first, the testers have something to start testing immediately. The programmers have also tested their design and code integration steps and so have a solid idea of how things will work when the whole feature is complete.

Testing Considerations While Prioritizing

It is important that the team understands the big picture or theme. In our example, team members know the stories for shipping outside the continental United States will come later. This knowledge may affect how they implement the first story. This doesn’t mean they have to plan for every eventuality, but if they know they need more shipping options, they may implement a drop-down list rather than a basic text field. No need to make more work or rework than necessary.

During release planning, we also consider the relative risk of the stories. If certain stories have many unknowns, it might be best to include them in an early iteration, so there’s time to recover if a story “blows up” and takes much more time than estimated. The same may apply to a story which, if not completed or implemented incorrectly, would have a costly negative impact. Scheduling it early will leave more time for testing.

If new technology or software is needed, it might be good to learn it by developing a straightforward story and plan more difficult ones for later iterations. This new technology may or may not affect your test automation. You may want more time to check out the impact. If the features are all brand new and the team needs more time to understand how they should work, plan to do less than your average velocity for the first iteration. That way, you’ll have more time to write tests that will correctly guide development. Identify risks and decide what approach makes the most sense from a testing perspective as well as a development perspective. This is one of the reasons it is important to include the whole team in the planning sessions.

Looking at the stories from a testing viewpoint is essential. This is where testers add the most value. The team needs to develop in small, testable chunks in order to help decide what stories are tentatively planned for which iteration. The key here is testable. Many new agile teams think small chunks means doing all of the database work first, or all of the configuration stuff. Testable doesn’t necessarily mean it needs a GUI either. For example, the algorithm that calculates shipping cost is an independent piece of code that can be tested independently of any user interface but requires extensive testing. That might be a good story for the first iteration. It can be tested as free-standing code and then later tested in combination with the UI and other parts of the system.

The testers may lobby for getting an end-to-end tracer bullet through the code quickly, so they can build an automation framework, and then flesh it out as the story development proceeds. If there are stories that present a big testing challenge, it might be good to do those early on. For example, if the release includes implementing a new third-party tool to create documents from templates and dynamic data, there are many permutations to test. If the team is unfamiliar with the tool, the testers can ask the team to consider doing those stories in the first iteration of the release.

What’s in Scope?

Agile teams continually manage scope in order to meet business deadlines while preserving quality. High-value stories are the first priority. Stories that are “nice-to-haves” might be elbowed out of the release.

Lisa’s Story

Our team’s customers list their stories in priority order and then draw a line between the stories that must be done before the release can occur, and the ones that could safely be put off. They call the less important stories “below the line,” and those stories may never get done.

For example, when we undertook the theme to allow retirement plan participants to borrow money from their retirement accounts, there was a “below the line” story to send emails to any participants whose loans are changing status to “pending default” or “default.” When the loan is in “default” status, the borrower must pay taxes and penalties on the balance. The email would be extremely helpful to the borrowers, but it wasn’t as important to our business as the software to request, approve and distribute loans, or process loan payments.

The email story didn’t make it into the release. It wasn’t done until more than two years later, after enough complaints from people who didn’t know their loans were going into default until it was too late.

—Lisa

Janet worked with a team whose customers were under the misplaced assumption that all of the features would get into their release and that when they were prioritizing, they were just picking which stories got done first. When the rest of the team realized the misunderstanding, they also implemented the idea of stories above and below the line. It helped to track progress as well as make the stories that were dropped below the line very visible.

Deadlines and Timelines

Many domains revolve around fixed dates on the calendar. Retail businesses make most of their profit during the holiday season. An Internet retail site is smart to have all new features implemented by October 1. Implementing a new feature close to the peak buying period is risky. Lisa’s company’s customers must complete government-required tasks during certain periods of the year. When it’s too late for a feature to get released this year, it often gets put off for the next year, because more urgent priorities must be addressed. Regulatory changes have specific timelines, and organizations have no choice about the timeline.

Janet’s Story

While working on this book, I was planning a release with my team at WestJet. We had several possible stories and worked with the customers to decide what the release would look like. We had one regulatory change that was light work for the programmers, but heavy for the testers. It needed to be in production by a certain date, so the other stories we were considering for the release took that into consideration.

We decided to create a small maintenance release with just that one major feature, along with a few bugs from the backlog so the release of the regulatory change would not be jeopardized. While the testers completed their testing, the rest of the team started some of elaboration stories for the next release.

An alternative plan could have been that the programmers chip in and help test and fit in more features. However, the whole team decided that this plan would work the best with the least amount of risk.

—Janet

Focus on Value

It’s rather easy for a team to start discussing a complex story and lose sight of what value the features actually deliver. Release planning is the time to start asking for examples and use cases of how the features will be used, and what value they’ll provide. Drawing flowcharts or sample calculations on the whiteboard can help pinpoint the core functionality.

Lisa’s Story

Our product owner wrote a story to provide a warning if an employer overrides the date a participant becomes eligible to contribute to a retirement account after the participant has already made contributions.

The warning needed to be incorporated into the legacy UI code, which didn’t easily accommodate it. The team discussed how it might be implemented, but every option was fairly costly. Not only would coding be tricky, but a lot of time was needed to test it adequately and update existing automated tests. This feature wouldn’t provide much value to the business, just a bit of help to the end users. The release was already pretty close to the limit on features.

One of the programmers suggested providing a report of participants who met the criteria so the plan administrators could simply call the employers who may need to make corrections. The report story was much smaller than the warning story, could easily fit into the release, and was acceptable to the customer.

—Lisa

There is no guarantee that these initial “guesstimates” at what will be in a given release will hold up over time. That is why customers needs to understand their priorities, take checkpoints at the end of every iteration, and reevaluate the priorities of remaining stories.

System-Wide Impact

One of our jobs as testers is to keep the big picture in mind. The agile tester thinks about how each story might affect the system as a whole, or other systems that ours has to work with. For example, if the toy warehouse makes a change to its inventory software, and the new code has a bug that overstates the number of items in stock, the website might sell more of the hot new doll than there are available, disappointing thousands of children and their parents at Christmas. When risk is high, listing areas of the system that might be affected for a theme or group of stories might be a worthwhile exercise even during release planning.

We talked about the “ripple effects” in Chapter 8, “Business Facing Tests that Support the Team.”

Contact points between our system and that of partners or vendors always merit consideration. Even a minor change to a csv or

Figure 15-5 shows a simplified diagram of a new system that touches many pieces of the existing system. Different tools might be needed to test the integrations.

Figure 15-5 System impacts

Testers who have worked with some of the other systems or understand what testing needs to happen on those systems can offer valuable insight into the impact of a new story. Often, stories will need to be delayed until a future release if the impact has not been explored. This is a good time to recall previous releases that didn’t end so well.

Third-Party Involvement

Working with vendor tools, partners, or other contractor teams on a big project complicates release planning. If anyone outside your team is responsible for some part of the project, that’s one piece that’s out of your control. If you need to coordinate with others, including possible new users of the system, it’s best to start early.

Lisa’s team has written several interfaces to allow users to upload data to their systems. In each case, they had to get the proposed file format out to the users early to make sure it would work for them. Other projects involved sending data to partners or vendors. These required extra planning to arrange testing with their test systems and getting their feedback on whether data was valid and correctly formatted.

If you’re using a third-party product as part of your solution, you might assume it has been tested, but that might be a poor assumption. You will need to budget extra time to test your application in conjunction with the vendor software. If there’s a problem in the other company’s software, it might take a long time to resolve. Lisa’s team uses third-party software for critical tasks such as document creation. If a theme includes modifying or creating new documents, they plan extra time to upgrade the software if needed, and extra time for testing in case fixes are needed. If possible, bring the third-party software into the project early, and start end-to-end testing. The more you can work with the interface, the better off you’ll be.

Other third-party software that we often forget about until it’s too late is our own testing environments. Sometimes a team will incorporate new code that takes advantages of new features in their chosen language. For example, if team members are using AJAX or JavaScript, they may need to upgrade the software development kit they’re using. This means that a team will have to upgrade its production runtime environment as well, so take that into consideration and test early.

Clients or partners might have concerns about a release that isn’t within your own team’s scope. Lisa’s team was once prevented at the very last minute from releasing a new feature because a partner didn’t have time to okay the change with their legal advisors. The programmers had to quickly devise a way to turn the functionality off without requiring extensive additional testing. Interestingly, partners who aren’t using agile development sometimes have trouble meeting their own deadlines. They might be unprepared when your team meets the deadline.

Janet’s Story

I worked on a project to implement a feature that required a new piece of hardware for scanning a new 2D bar code. The team decided to implement in stages because it was not known when the scanners would be available for full testing, but the customer wanted the code ready for when the scanners arrived.

The initial phase was programmer-intensive because there was a lot of research to be done. After they determined how they would implement the feature, the story was created to add it into the code. However, we knew we couldn’t thoroughly test it until the scanners were available. The code was ready to test, but instead of backing it all out, we only needed to worry about testing that the feature could be turned off for the release. The next release would require more testing, but only if the scanners were available. The testing of the story was kept in the product backlog so we would not forget to do it.

—Janet

If you’ll be working with other teams developing different components of the same system, or related systems, budget time to coordinate with them. It’s a good idea to designate a member from each team to coordinate together.

Release planning is the time to identify extra roles you need on your team, additional resources, and time needed for out-of-the-ordinary circumstances.

Test Planning

We can’t expect to plan the iterations in a release at a detailed level. We can get an idea of the theme’s steel threads, prioritize stories, and make a guess at what stories will be in which iteration. Detailed test planning needs to wait for iteration planning. Still, we need to think about testing at a high level, and try to budget enough time for it. We might even take time separately from the release planning meeting to strategize our testing for the release. In Chapter 8, Business-Facing Tests that Support the Team, we mentioned one of the perils of agile testing: “forgetting the big picture.” Test Planning will help you with that problem.

Chapter 8, “Business-Facing Tests that Support the Team,” explains how to identify steel threads or thin slices in a story or theme.

Where to Start

During release planning, it’s helpful to know the business conditions of satisfaction for each story or high-level user acceptance test case. When stories need clarification, agile testers ask for examples. At this stage, examples will be high-level, covering just the basics, but enough to be able to size and prioritize the story. Drawing flowcharts or writing calculations on the whiteboard and discussing them helps us identify project-specific testing issues.

At a minimum, the team needs to understand the top-priority stories that are scheduled to be performed first. Lightweight planning might involve only looking at those core stories with the understanding that more time will be needed for defining additional tests.

As we get a sense of which stories will probably be included in the release, we can start thinking about the scope of the testing. What assumptions have been made that might affect testing? Use of third-party software, such as the example of using a shipping company’s shipping calculation API, affects test planning. Are there any unusual risks in this release that will impact testing? If we have stories to implement batch jobs, and we’ve never had any batch processing in the system before, there are probably new frameworks that impact testing. We need to budget time to learn them.

Why Write a Test Plan?

In release planning, we talk about the purpose of the release, what’s in scope, and what assumptions we’re making. We do some quick risk analysis and plan our test approach to address those risks. We consider automation and what we need for test environments and test data. We certainly want to identify milestones and deliverables. Hmmm, this is starting to sound like . . . a test plan!

If, like ourselves, you spent time working in a traditional waterfall environment, you might have wasted time writing bulky test plans that nobody read and nobody bothered to maintain. In agile development, we want our test plans to serve our needs. Your customer might require a test plan for each release for compliance reasons. Even if it’s not a required deliverable, it can be useful. Keep it concise and lightweight. It only has to serve your purposes during this release. Address the testing issues that are specific to this release or project. Include your risk analysis and identify assumptions. Outline the critical success factors that your customer has identified. Think about what people need to know related to testing, and remove anything extraneous.

See Chapter 5, “Transitioning Typical Processes,” for more about test plans and test strategies.

Even if you don’t create a formal test plan, be sure you have made note of all these different testing factors involved in the release. You’ll want to keep them in mind during every iteration planning session. The biggest benefit in test planning is the planning itself. It allows you to consider and address issues such as test data requirements, infrastructure, or even what test results are required. Test planning is a risk mitigation strategy. Let’s consider some of these issues.

Types of Testing

In Part III, we covered the four quadrants of testing and talked about all of the different types of testing you can do during your project. Release planning is a good time to consider these different needs. Do you need to plan to bring in a load test tool, or will there be the need to build some kind of specialty test harness?

It could be that your next release is just an extension of your last, and you will just carry on creating your examples, automating your story tests, and doing the rest of the testing as you’ve been doing. You are one of the lucky ones. For those of you who are starting a brand new project with no previous processes in place, now is the time to consider what testing you will need. We don’t mean you have to decide how to test each story, but look at the big picture and think about the quadrants. Will you need to plan for a special UAT, or will the iteration demos be enough? It is important to raise these issues early so the team can plan for them.

Infrastructure

While you are doing your test planning, you need to consider your infrastructure. Infrastructure can mean your continuous integration setup, test environments, and test database. It can mean how you promote your builds to your test environments. It might mean your test lab, if you have one, or having a separate server to run all your automation tests. These are generally pieces of infrastructure that need some lead time to get in place. This is the time to make a plan.

Lisa’s Story

Some types of testing might require extra effort. My team had a tool to do performance testing and some scripts, but we lacked a production-style environment where we could control all of the variables that might affect performance. For example, the test database was shared by testers, programmers, and two build processes. Slower performance might simply mean someone was running database-intensive tests. We used our staging environment to get a baseline, but it was missing some of the components of production. We set a six-month goal to acquire hardware and software for a proper test environment and get it set up. We wrote a task card or two each iteration to establish the environment step by step.

—Lisa

Whatever your needs are, make sure you understand them and can plan for what you need. If you don’t have the right infrastructure, then you will waste time trying to get it together and cause a bottleneck in mid-iteration.

Test Environments

As we look at the types of features in the next release, we might see the need for a whole new test environment. Think about specialized test environments you may need as well. Will you need more tools? Do you need to expand your test lab so that you can test with different browsers and operating systems? This is the time to think about all testing considerations.

If you’re planning your first release, test environments are a key consideration. You might need a story or iteration just to set up the infrastructure you need. We’ve started more than one project where the only place we could test was the development environment. We found that doesn’t work very well, because the environment is never stable enough for effective testing.

Just as programmers have their own sandboxes to work and test in, it works well if each tester has that same availability and control. We recognize that not all applications lend themselves to this, but at the very least, you need to know what build you’re testing. You also need test data that others will not walk over with their tests. If you don’t have a testing sandbox that’s under your own control, take time to plan what you need to establish for your test environments. Brainstorm with your team about how you can obtain the necessary hardware and software. It might take time, so develop a Plan B for getting something done while waiting for the infrastructure you need.

If you’re working on a large system, you may have to queue up along with other teams to get time on a test or staging environment that includes all of the various pieces of software with which yours must work. This staging environment should mimic your production system as much as possible. If your organization doesn’t have someone responsible for creating environments, your team might need extra roles dedicated to obtaining the test environments you need. These roles might involve working with other teams as well. Release planning is the time to consider all of these testing infrastructure requirements.

Test Data

Release or theme planning is also a good time to think about what test data you might need during the project.

Using test data that closely resembles real data is generally a good practice. Plan for the data you need. We’ve had the opportunity in several organizations to use a copy of production data. Real data provides a good base for different scenarios for exploratory testing. Production data may need to be “scrubbed” before it’s used for testing in order to remove any sensitive information such as identification or bank account numbers. The data needs to be altered to hide the original values but remain valid so that it doesn’t violate database restrictions. Because it takes time for database experts to port production data to a test environment, be sure they’re included in your planning.

Janet’s Story

In one of the organizations I was working with, we used two different baseline test data schemes. For our individual test environments, we used Fit fixtures to load predefined data. We tried to make this data as close to production as possible, but we also seeded it with some very specific test data. Every time we checked out a new version of code, we were able to reload a base set of data. In this way, we also tested the database schema as well to see if anything had changed.

For our more stable test environment where we wanted data persisted, we used the data migration scripts that the programmers developed as they made database changes. These migration scripts were eventually used for the initial cut over from production and by then we were pretty certain they were correct.

—Janet

Enlist your customers’ support in obtaining meaningful test data. If you’re working on a story that involves sending a file to a third-party vendor, your business expert can find out what data the vendor expects in the file. Lisa’s team developed features to allow retirement plan brokers to offer their customers portfolios of mutual funds. They asked the product owner to provide samples of portfolios, including a name, description, and set of funds for each. This helped them test with realistic data.

Test data tends to get stale and out of date over time. Older data, even if it came from production, may no longer accurately reflect current production data. A “passing” test using data that’s no longer valid gives a misleading sense of confidence. Continually review your test data needs. Refresh data or create it using a new approach, as needed.

Test data requirements vary according to the type of testing. Regression tests can usually create their own data or run against a small representational set of data that can be refreshed to a known state quickly. Exploratory testing may need a complete replica of production type data.

Chapter 14, “An Agile Test Automation Strategy,” explores different approaches to obtaining test data

Test Results

Different teams have different requirements for test result reporting. Think about how you are going to report test results at this stage of the game so that you can do so effectively when the time comes to do the actual reporting. Your organization may have audit compliance requirements, or maybe your customer just wants to know how you tested. Understand your needs so that you can choose the approach that is right for your team.

There are many ways to report test results. There are vendor tools that will record both automated and manual results. Your team may find a way to persist the results from tools such as Fit, or you may just choose to keep a big visible manual chart.

The approach that a few teams have taken is to create home-grown test result applications. For example, a simple Ruby application written with Ruby on Rails for the database or a MySQL database with a PHP front end can make a very simple but easy-to-use test management system.

A tool such as this can be very simple or can include added complexity such as the capability to categorize your tests. The important thing is the test results. If your automated tests record their pass or fail result along with the error, you have some history to help determine fragility of the test.

Your team can configure your automated build process to provide test results from each build, by email, or a feedback utility or web interface that team members can view online. Results over time can be summarized in a variety of formats that make progress visible. One of Lisa’s teams produced a daily graph of tests written, run, and passing that was posted in the team’s work area. Another produced a daily calendar with the number of unit tests passing every day. Even simple visual results are effective.

We talk about some of the metrics you can use later in this chapter.

Test Plan Alternatives

We’ve talked about why to test plan and what you should consider. Now we talk about some of the alternatives to the heavy test plans you may be used to. Whatever type of test plan your organization uses, make it yours. Use it in a way that benefits your team, and make sure you meet your customer’s needs. As with any document your team produces, it should fulfill a purpose.

Lightweight Test Plans

If your organization or customer insists on a test plan for SOX compliance or other regulatory needs, consider a lightweight test plan that covers the necessities but not any extras. Do not repeat items that have already been included in the Project Plan or Project Charter. A sample Test Plan might look something like the one shown in Figure 15-6.

Figure 15-6 Sample Test Plan

A test plan should not cover every eventuality or every story, and it is not meant to address traceability. It should be a tool to help you think about testing risks to your project. It should not replace face-to-face conversation with your customer or the rest of your team.

Using a Test Matrix

Janet uses release planning to work with the testers and customers to develop a high-level test matrix. A test matrix is a simple way to communicate the big picture concerning what functionality you want to test. It gives your team a quick overview of the testing required.

A test matrix is just a list of functionality down the side and test conditions across the top. When thinking about test conditions and functionality, consider the whole application and any impact the new or changed functionality might have on the rest of the application. Testers sitting with customers and thinking about test conditions is what is important.

It can also be a mechanism to track coverage and can be as detailed as you like. A high-level test matrix can be used by the team to show the customer team or management what has been tested already and what is left. A more detailed test matrix can be used by the team to show what is planned for testing and track the progress of the testing. After the matrix has been created, it becomes easy to fill in the squares when testing is done. Keep it simple. Because we like big visible charts that are easy to read, we recommend colors that mean something to your team. For example, green (G) means testing is done and the team is happy with it, while yellow (Y) might mean some testing has been done but more exploratory testing is needed if there is time. Red (R) means something is broken. A white square means it hasn’t been tested yet, and a gray (not applicable) square means it doesn’t need to be tested.

Let’s look at an example. We have a small release we want to put out that calculates shipping costs. In Figure 15-7, different pieces of functionality are represented on one axis, and properties of the shipment are represented on the other. Individual cells are color-coded to show which cases are tested and which need more attention. All of the cells for “<= 2 lbs” are finished, the top three cells for > 4 lbs are done but need more exploratory testing, and the “Ship to Alaska”/“>4 lbs” cell denotes a possible issue.

Figure 15-7 A sample test matrix

Janet’s Story

I had an unexpected side effect from using a test matrix in one project I was on. The customers and testers put the test matrix together, and had thought of all affected functionality for the project and the high-level test conditions they would need. As expected, the act of planning brought a lot of issues out that would have been missed until later.

When they hung the matrix on the wall in their team area, Dave, the developer team lead, expressed an interest. One of the testers explained the matrix to him, and I was surprised when he said it was very useful for them as well. Dave said “I didn’t know that this functionality would affect this area. We need to make sure our unit tests touch on this as well.”

Looking back on this, I shouldn’t have been surprised, but I had never had that experience with the programmers before.

—Janet

A test matrix is a very powerful tool and can be used to help address traceability issues if your team has those problems. Think about what makes sense for your team and adapt it for your team and what makes sense to you.

Test Spreadsheet

Janet has also seen a spreadsheet format used with some success. For example, at WestJet, the first tab in a workbook was a high-level list of functionality that existed in the application. For each row, the team determined if the project affected that piece of functionality. If so, they gave a rating of the expected impact. After the impact of the changes had been determined, decisions about test environments, test data, or UAT could then be made.

Tabs were used for risks and assumptions but could be used for anything your team may need. A flexible format such as a spreadsheet means you can tailor it to work for you.

This information can be used in a number of different ways. It can be used to determine where to concentrate your exploratory testing efforts, or maybe to help create a high-level test matrix to make sure you touch on all of the areas during your testing.

A Whiteboard

If your team is informal and has small releases, any kind of documentation may be too much. Sometimes it’s enough to list the risks and assumptions on a whiteboard or on index cards. Janet has used a whiteboard to manage risks, and it worked quite well. If a risk actually became an issue, the result was documented and crossed off. It was easy to add new risks and mitigation strategies, and the list was visible to the whole team. This could also be done on a wiki page.

We cannot stress enough that you need to know your team and its needs.

Automated Test List

Sometimes you may be required to present more information to your customers, such as a list of test cases. If your team has a tool from which you could extract a list of test case names, you could provide this list easily to anyone who needed it. This would present more of a traditional type detailed test plan but wouldn’t be available until after tests were actually written. We don’t recommend spending any time on this because we don’t see added value, but sometimes this list may be required for risk assessment or auditability.

Preparing for Visibility

If your team is just getting started with agile development, make sure you have necessary infrastructure in place for your early iterations. You may change the way you are tracking progress as you go along, and your retrospectives will help you bring these issues to light. If you’re having problems completing the work planned for each iteration, maybe you need more visible charts or visual aids to help you gauge progress and make mid-iteration adjustments. Do your customers have some way to know how the iteration is progressing and which stories are done? Take time before the each iteration to evaluate whether you’re getting the right kind of feedback to keep track of testing.

Tracking Test Tasks and Status

The effective agile teams we know all follow this simple rule: “No story is done until it’s tested.” This rule can be expanded to say that not only must the story be tested, the code must be checked in, it must have automated tests that are run by a continual build process, it must be documented, or whatever your team’s “doneness” criteria are. At any time during an iteration, you need to be able to quickly assess how much testing work remains on each story, and which stories are “done.” Story or task boards are ideal for this purpose, especially if they use color-coding to denote test tasks vs. development and other types of tasks. Cork boards, steel sheets with magnets, poster-sized sticky notes, or whiteboards all work fine. Give each story its own row, and order them by priority. Have columns for “to do,” “work in progress,” “verify,” and “done.”

Janet’s Story

I started with team members who had been doing agile for a few months with only a couple of programmers and one tester. They had been using XPlanner to track their tasks and stories, and it was working ok for them. At the same time I came on board, a couple of new programmers were added, and the stand-ups became less effective; the team was not completing the stories it had planned. I suggested a storyboard, and although they were skeptical about keeping two sets of “tasks,” they said they would try it.

We took an open wall and used stickies to create our story board. We started having stand-ups in front of the story board and our discussion became more specific. It provided a nice visible way of knowing when the tasks were done and what was left to do. After a couple of months, the team grew again and we had to move the story board into an office. We also moved our stand-ups and our test result charts there. However, the constant visibility was lost, and programmers and testers stopped moving their tasks.

We had to reevaluate what we wanted to do. One size does not fit all teams. Make sure you plan for what is right for your team.

—Janet

Some teams use different colored index cards for the different types of tasks: green for testing, white for coding, yellow and red for bugs. Other teams use one card per development task, and add different colored stickers to show that testing is in progress or show that there are bugs to resolve. Use any method that lets you see at a quick glance how many stories are “done,” with all coding, database, testing, and other tasks completed. As the iteration progresses, it’s easy to see if the team is on track, or if you need to pull a story out or have programmers pitch in on testing tasks.

Janet’s Story

Our story board (shown in Figure 15-8) wasn’t very big, and we didn’t have a lot of wall space to expand to have the regular column-type task board. Instead, we decided to use stickers to designate the status.

Figure 15-8 Example story board

White cards, such as those shown in the first row of Figure 15-8, were regular tasks, blue cards designated technical stories such as refactoring or spikes, and pink cards, shown toward the right-hand side of the board as the darkest color, were bugs that need to be addressed. It is easy to see that this picture was taken at the beginning of an iteration because there are no colored circles on each card. In the top right-hand corner, you can see the legend. Blue stickers meant it has been coded, green would indicate done (tested), and red meant the task has been deemed not completed or a bug was rejected as not fixed. As a task or story was completed (i.e., green sticker), it was moved to the right of the board.

—Janet

Lisa’s Story

For more than four years, our story board was a couple of sheets of sheet metal, painted in company colors, using color-coded index cards attached to the board with magnets. Figure 15-9 shows a picture of it early in an iteration. Our task cards were also color-coded: white for development tasks, green for coding tasks, yellow and red for bugs, and striped for cards not originally planned in the iteration. The board was so effective in indicating our progress that we eventually stopped bothering with a task burndown chart. It let us focus on completing one story at a time. We also used it to post other big visible charts, such as a big red sign showing the build had failed. We loved our board.

Figure 15-9 Another sample story board. Used with permission of Mike Thomas. Copyright 2008.

Then, one of our team members moved overseas. We tried using a spreadsheet along with our physical story board, but our remote teammate found the spreadsheet too hard to use. We tried several software packages designed for Scrum teams, but they were so different from our real story board that we couldn’t adjust to using them. We finally found a product (Mingle) that looked and worked enough like our physical board that everyone, including our remote person, could use it. We painted our old story board white, and now we can project the story board on the wall during stand-up meetings.

—Lisa

Distributed teams need some kind of online story board. This might be a spreadsheet, or specialized software that mimics a physical story board as Mingle does.

Communicating Test Results

Earlier, we talked about planning how to track test results. Now we want to talk about effectively communicating them. Test results are one of the most important ways to measure progress, see whether new tests are being written and run for each story, and whether they’re all passing. Some teams post big visible charts of the number of tests written, run, and passed. Others have their build process email automated test results to team members and stakeholders. Some continuous integration tools provide GUI tools to monitor builds and build results.

We’ve heard of teams that have a projector hooked up to the machine that runs FitNesse tests on a continuous build and displays the test results at all times. Test results are a concrete depiction of the team’s progress. If the number of tests doesn’t go up every day or every iteration, that might indicate a problem. Either the team isn’t writing tests (assuming they’re developing test-first), or they aren’t getting much code completed. Of course, it’s possible they are ripping out old code and the tests that went with it. It’s important to analyze why trends are going the wrong way. The next section gives you some ideas about the types of metrics you may want to gather and display.

However your team decides they want to communicate your progress, make sure you think about it up front and everyone gets value from it.

Release Metrics

We include this section here, because it is important to understand what metrics you want to gather from the very beginning of a release. These metrics should give you continual feedback about how development is proceeding, so that you can respond to unexpected events and change your process as needed. Remember, you need to understand what problem you are trying to solve with your metrics so that you can track the right ones. The metrics we talk about here are just some examples that you may choose to track.

Number of Passing Tests

Many agile teams track the number of tests at each level: unit, functional, story tests, GUI, load, and so on. The trend is more important than the number. We get a warm fuzzy feeling seeing the number of tests go up. A number without context is just a number, though. For example, if a team says it has 1000 tests, what does that mean? Do 1000 tests give 10% or 90% coverage? What happens when code that has tests is removed?

Tracking the number of tests written, running, and passing at a story level is one way to show a story’s status. The number of tests written shows progress of tests to drive development. Knowing how many tests aren’t passing yet gives you an idea of how much code still needs to be written.

After a test passes, it needs to stay “green” as long as the functionality is present in the code. Graphs of the number of tests passing and failing over time show whether there’s a problem with regression failures and also show the growth of the code base. Again, it’s the trend that’s important. Watch for anomalies.

These types of measurements can be reported simply and still be effective.

Lisa’s Story

My team emails a color-coded calendar out every day showing whether the “full build” with the full suite of regression tests passed each day (see Figure 15-10). Two “red” days in a row (the darkest color) are a cause for concern and noticed by management as well as the development team. Seeing the visual test results helps the organization pull together to fix the failing tests or any other problems causing the build to not run, such as hardware or database issues.

—Lisa

Figure 15-10 Full build result email from Lisa’s team

There are different ways to measure the number of tests. Choose one and try to stay consistent across the board with all types of tests, otherwise your metrics may get confusing. Measuring the number of test scripts or classes is one way, but each one may contain multiple individual test cases or “asserts,” so it may be more accurate to count those.

If you’re going to count tests, be sure to report the information so that it can be used. Build emails or build status UIs can communicate the number of tests run, passed, and failed at various levels. The customer team may be content to see this information only at the end of each sprint, in the sprint review, or an email.

Whatever metrics you choose to gather, be sure the team buys into them.

Janet’s Story

I started a new contract with a team that had been doing agile for a couple of years, and they had developed a large number of automated functional tests. I started keeping track of the number of tests passing each day. The team didn’t see a problem when the trending showed fewer and fewer tests were passing. The unit tests were maintained and were doing what they were supposed to do, so the team felt confident in the release. It seemed this happened with every release, and the team would spend the last week before the release to make all of the tests pass. It was costly to maintain the tests, but the team didn’t want to slow down to fix them. Everyone was okay with this except me.

I did not see how fixing the tests at that late date could ensure the right expected results were captured. I felt that we ran the risk of getting false positives.

At the start of the next release cycle, I got the team to agree to try fixing the tests as they broke. It didn’t take long for the team to realize that it wasn’t so tough to fix the tests as soon as we knew they were broken, and we found a lot of issues early that hadn’t usually been caught until much later. The team soon set a goal of having 95% of the tests passing at all times.

We also realized how brittle the tests were. The team made a concerted effort to refactor some of the more complex tests and eliminate redundant ones. Over time, the number of high-level tests was reduced, but the quality and coverage was increased.

We started out measuring passing rates, but we ended up with far more.

—Janet

Don’t get so caught up in the actual measurements that you don’t recognize other side effects of the trending. Be open to adjusting what you are measuring if the need is there.

Code Coverage

Code coverage is another traditional metric. How much of our code is exercised by our tests? There are excellent commercial and open source code coverage tools available, and these can be integrated into your build process so that you know right away if coverage has gone up or down. As with most metrics, the trend is the thing to watch. Figure 15-11 shows a sample code coverage report.

Table 15-11. Sample code coverage report from Lisa’s team. “Ghidrah” is the new architecture; “Whitney” is the legacy system.

Figures 15-12 and 15-13 are two examples of trends that work together. Figure 15-12 shows a trend of the total number of methods each iteration. Figure 15-12 is the matching code coverage. These examples show why graphs need to be looked at in context. If you only look at the first graph showing the number of methods, you’ll only get half the story. The number of methods is increasing, which looks good, but the coverage is actually decreasing. We do not know the reason for the decreased coverage, but it should be a trigger to ask the team, “Why?”

Figure 15-12 Number of methods trend

Figure 15-13 Test coverage

Remember that these tools can only measure coverage of the code you’ve written. If some functionality was missed, your code coverage report will not bring that to light. You might have 80% code coverage with your tests, but you’re missing 10% of the code you should have. Driving development with tests helps avoid this problem, but don’t value code coverage statistics more than they deserve.

Know What You Are Measuring

Alessandro Collino, a computer science and information engineer with Onion S.p.A. who works on agile projects, told us about an experience where code coverage fell suddenly and disastrously. His agile team developed middleware for a real-time operating system on an embedded system. He explained:

A TDD approach was followed to develop a great number of good unit tests oriented to achieve good code coverage. We wrote many effective acceptance tests to check all of the complex functionalities. After that, we instrumented the code with a code coverage tool and reached a statement coverage of 95%.

The code that couldn’t be tested was verified by inspection, leading them to declare 100% of statement coverage after ten four-week sprints.

After that, the customer required to us to add a small feature before we delivered the software product. We implemented this request and applied the code optimization of the compiler.

This time, when we ran the acceptance tests, the result was disastrous; 47% of acceptance tests failed, and the statement coverage had fallen down to 62%!

What happened? The problem turned out to be due to enabling compiler optimization but with an incorrect setting. Because of this, a key value was read once as the application started up and was stored in a CPU register. Even when the variable was modified in memory, the value in the CPU register was never replaced. The routine kept reading this same stale value instead of the correct updated value, causing tests to fail.

Alessandro concludes, “The lesson learned from this example is that the enabling of the compiler optimization options should be planned at the beginning of the project. It’s a mistake to activate them at the final stages of the project.”

Good metrics require some good planning. Extra effort can give you more meaningful data. Pierre Veragen’s team members use a break-test baseline technique to learn if their code coverage metric is meaningful. They manually introduce a flaw into each method and then run their tests to make sure the tests catch the problem. Some tests just make sure the code returns some value, any value. Pierre’s team makes sure the tests return the correct value. In this way, they can determine whether their test coverage is good enough.

Code coverage is just one small part of the puzzle. Use it as such. It doesn’t tell you how good your tests are but only if a certain chunk of code was run during the test. It does not tell you if different paths through the application were run, either. Understand your application and try identifying your highest risk areas, and set a coverage goal that is higher for those areas than for low-risk areas. Don’t forget to include your functional tests in the coverage report as well.

Defect Metrics

As your team sets goals related to defects, use appropriate metrics to measure progress toward those goals. There are trends that you will want to monitor for the whole release, and there are ones that are iteration-specific. For example, if you’re trying to achieve zero defects, you may want to track the open bugs at the end of each iteration, or how many bugs were found after development but before release. Most of us are interested in knowing how many defects have been reported after the code is in production, which is something completely different altogether. These issues will tell you after the fact how well your team did on the last release, but not how well you are doing on the current release. They may give you some indication of what processes you need to change to reduce the number of defects. Lisa’s team is more concerned with production defects found in the “new” code that was rewritten in the new architecture. They’re working hard to produce this new code with zero defects, so they need to know how well they’re doing. They expect that bugs will be found fairly often in the legacy system, where only the most critical functionality is covered by automated GUI smoke tests, and there are few automated unit and behind-the-GUI tests.

Knowing the defect rate of legacy code might be good justification for refactoring or rewriting it, but the team’s top priority is doing a good job with the new code, so they group bugs by “new” and “old” code, and focus on the “new” bugs.

Make sure your bug database can track what you want to measure. You may have to make some changes in both the database and your process to get the data you need. For example, if you want to measure how many defects were found in production after a release, you have to make sure you have environment and version as mandatory fields, or make sure that people who enter bugs always fill them in.

More on defect tracking systems can be found in Chapter 5, “Transitioning Traditional Processes.”

Because defect tracking systems are often used for purposes besides tracking bugs, be sure not to muddle the numbers. A request for a manual update to the database doesn’t necessarily reflect an issue with the existing code. Use your defect tracking tool properly to ensure that your metrics are meaningful.

Lisa’s Story

Periodically evaluate the metrics you’re reporting and see if they’re still relevant. Figure 15-14 shows two defect reports that Lisa’s team used for years. When we first transitioned to agile, managers and others looked at these reports to see the progress that resulted from the new process. Four years later, our ScrumMaster found that nobody was reading these reports anymore, so we quit producing them. By that time, rates of new defects had reduced dramatically, and nobody really cared about the old defects still hanging about in the legacy code.

—Lisa

Figure 15-14 Sample defect reports used (and no longer used) by Lisa’s team

Release planning is a good time to evaluate the ROI of the metrics you’ve been tracking. How much effort are you spending to gather and report the metrics? Do they tell you what you need to know? Does the code you release meet your team’s standards for internal quality? Is the code coverage percentage going up? Is the team meeting its goals for reducing the number of defects that get out to production? If not, was there a good reason?

Metrics are just one piece of the puzzle. Use your release, theme, or project planning meetings to refocus on delivering business value when the business needs it. Take some time to learn about the features you’re about to develop. Don’t get caught up with committing to your plans—the situation is bound to change. Instead, prepare for doing the right activities and getting the right resources in time to meet the customers’ priorities.

Summary

As your team puts together its plan for a new theme or release, keep the main points of this chapter in mind.

When sizing a story, consider different viewpoints, including business value, risk, technical implementation, and how the feature will be used. Ask clarifying questions, but don’t get bogged down in details.

Testers can help identify the “thin slice” or “critical path” through a feature set to help prioritize stories. Schedule high-risk stories early if they might require extra testing early.

The size of testing effort for a story helps determine whether that story is in scope for the release.

Testers can help the team think about how new stories will impact the larger system.

Plan for extra testing time and resources when features may affect systems or subsystems developed by outside teams.

As the team identifies the scope of the release, evaluate the scope of testing and budget enough time and resources for it.

Spend some time during release planning to address infrastructure, test environment, and test data concerns.

A lightweight, agile test plan can help make sure all of the testing considerations are addressed during the life of the release or project.

Consider alternatives to test plans that might be more appropriate for your team; test matrices, spreadsheets, or even a whiteboard may be sufficient.

Formal release planning may not be appropriate for your situation. In the absence of release planning, consider identifying and discussing at least the first few stories that should be done first.

Plan for what metrics you want to capture for the life of the release; think about what problem you are trying to solve and capture only those metrics that are meaningful for your team.