Agile Testing: A Practical Guide for Testers and Agile Teams

Chapter 14 An Agile Test Automation Strategy

As we explored each of the Agile Testing Quadrants in Part III, we gave examples of tools that can help those different testing efforts succeed. Many of those tools are for automating tests. As we described in the previous chapter, teams face plenty of obstacles in their quest for successful test automation. Better tools become available all the time, but the trick is to choose the right tools and learn to use them effectively. Test automation requires thoughtful investment and incremental improvement. In this chapter, we explain how you can apply agile values and principles to get traction in starting or improving your automation efforts.

An Agile Approach to Test Automation

Here you are, reading this chapter on how to get your test automation strategy working, maybe hoping for that silver bullet, or an answer to all your questions. We hate to disappoint you, but we need to tell you right up front, there is no silver bullet. There is no one answer that works for every team. Don’t lose heart though, because we have some ideas to help you get started.

First, we suggest approaching your automation problems as you would any problem. Define the problem you are trying to solve. To help you figure that out, we first talk about some basics of test automation and reintroduce some terms.

Automation Test Categories

In Part III, we introduced the Agile Testing Quadrants and talked about each quadrant and the purpose of the tests in each quadrant. In this section, we look at the quadrants in a different light. Let’s look carefully at the quadrants (see Figure 14-1).

Figure 14-1 Agile Testing Quadrants

You can see that we’ve labeled both quadrants that support the team (Q1 and Q2) as using automation. In Quadrant 4, the tools used for critiquing the product from a technology point of view also usually require automated tools. In Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” we discussed some of the tools that can be used for automating business-facing tests in the quest for supporting the team. In fact, the only quadrant that is not labeled as using automation is Quadrant 3—the business-facing tests that critique the product. However, as we discussed in Chapter 10, “Business-Facing Tests that Critique the Product,” tools may be useful for some of that testing. For example, automation can help set up test data and user scenarios, and analyze logged activity.

Use the quadrants to help you identify the different types of automation tools you might need for each project, even for each iteration. We find it helpful to go through each quadrant and make a checklist of what tools might be needed. Let’s say we’re about to redesign a UI. We look at Quadrant 1. How can it be coded test-first? Do we know how to unit test our presentation layer? Do we need a new tool to help with that? Now on to Quadrant 2. We’ll need to do some prototyping; should we just use paper, or should we plan a Wizard of Oz type activity? What tool will we use to create executable business-facing tests to guide development? Do we have regression test scripts that will need updating or replacing? We know that one of our Quadrant 3 activities will be usability testing. That takes some advance planning. We might want tools to help track the users’ activities so we can analyze them further. Thinking about Quadrant 4, we realize that we have load test scripts that use the old UI, so we have to budget time to update them for the new one.

See Chapter 8, “Business-Facing Tests that Support the Team,” for more about Wizard of Oz testing.

As we emphasized in Part III, “Using the Agile Testing Quadrants,” the order of quadrants doesn’t relate to the order in which we do the testing. As we make our checklist of tools needed for each type of test, we think about when we want to test so we know when to have our automation tools ready. For example, a team designing a new architecture would plan to do a spike and run scalability test against it as soon as possible. They will need to spend time during the first iteration of the project finding and implementing a performance test tool.

The quadrants help us figure out what tools we might need, but with so many different automation options at different levels, a strategy for where to do which types of testing and how to organize the tests is essential. To deliver value quickly and often, our automation efforts need a high ROI. The test pyramid helps us optimize our test investment.

Test Automation Pyramid

Figure 14-2 illustrates the “Test automation pyramid.” We like the version that Mike Cohn introduced, which shows the foundation layer made up of technology-facing unit and component tests. We recognize that many teams will struggle with this idea, because it seems the opposite of what many teams currently have. Many test teams have been taught the “V” model of testing, where activities such as component, system, and release testing are done in sequence after coding activities. Other teams have an inverted pyramid, with the majority of the tests in the functional or presentation layer.

Figure 14-2 Test automation pyramid

The agile test automation pyramid shows three different layers of automated tests. The lowest tier is the foundation that supports all of the rest. It’s mainly made up of robust unit tests and component tests, the technology-facing tests that support the team. This layer represents the bulk of the automated tests. They’re generally written in the same language as the system under test, using the xUnit family of tools. After a team has mastered the art of TDD, these tests are by far the quickest and least expensive to write. They provide the quickest feedback, too, making them highly valuable. They have the biggest ROI by far of any type of test.

In agile development, we try to push as many tests as possible to this layer. While business-facing tests tend to go in one of the higher levels, we implement them at the unit level when it makes sense. If they’re tests the customers don’t have to be able to read, and they can be coded much more quickly as unit tests, it’s a good option. Other types of technology-facing tests such as performance tests may also be possible at the unit level.

See Chapter 7, “Technology-Facing Tests that Support the Team” for more about unit and component tests.

The middle tier in the pyramid is the layer that includes most of the automated business-facing tests written to support the team. These are the functional tests that verify that we are “building the right thing.” The tests in this layer may include “story” tests, “acceptance” tests, and tests that cover larger sets of functionality than the unit test layer. These tests operate at the API level or “behind the GUI,” testing the functionality directly without going through the GUI. We write test cases that set up inputs and fixtures that feed the inputs into the production code, accept the outputs, and compare them to expected results. Because these tests bypass the presentation layer, they are less expensive to write and maintain than tests that use the interface.

See Chapter, 8, “Business-Facing Tests that Support the Team,” for more about business-facing tests that support the team.

We try to write them in a domain-specific language that the customers can understand, so they take more work than unit-level tests. They also generally run more slowly, because each test covers more ground than a unit test and may access the database or other components. The feedback they provide is not as quick as the unit-level tests, but it is still much faster than we could get operating through the user interface. Therefore, their ROI is not as high as the tests that form the base of the pyramid, but it’s higher than the top layer.

Fit and FitNesse are examples of tools used for the middle layer of the pyramid. Home-grown test harnesses that use spreadsheets or other business-friendly means for defining test cases are also common.

We have more about these tests in Chapter 8, “Business-Facing Tests that Support the Team,” and Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” where we discuss the business-facing tests that support the team and the tools that effectively capture these tests.

The top tier represents what should be the smallest automation effort, because the tests generally provide the lowest ROI. These tests are the ones done through the GUI, the ones that actually operate and manipulate the presentation layer. They are written after the code is completed, and so are usually written to critique the product and go directly to the regression suite.

These tests are traditionally more expensive to write, although there are new tools that help reduce the investment needed. Because components of the user interface tend to be changed often, these tests are much more brittle than tests that work at a functional or unit level. For example, just renaming HTML elements could cause a test script to fail. Operating through the user interface also slows these tests down, compared to tests in the lower levels of the pyramid that operate directly on production code. The tests in the top layer do provide important feedback, but a suite of GUI tests may take hours to run rather than the few minutes required for unit-level test suites. We want to minimize the number of tests at this layer, so they should only form the tip of the pyramid.

No matter how many automated tests they have, most systems also need manual testing activities, such as exploratory testing and user acceptance testing. We don’t want to forget these, so we’ve illustrated them with the little cloud at the tip of the pyramid. The bulk of our regression testing must be automated or our manual testing won’t give us a good return on investment either.

Patrick Wilson-Welsh [2008] adds a descriptive dimension to the test automation pyramid with a “three little pigs” metaphor. The bottom foundation layer is made of bricks. The tests are solid, and not vulnerable to the huffing and puffing of the Big Bad Wolf. The middle layer is made of sticks. They need rearranging more often than the brick layer to stay strong. The tests in the top layer are made of straw. It’s hard to get them to stay in place, and the wolf can easily blow them around. If we have too many tests made out of straw, we’re going to spend lots of time putting them back into shape.

Most new agile teams don’t start with this shape pyramid—it’s usually inverted, a left-over from previous projects. GUI test tools are often easier to learn, so teams start out with a lot of tests in their top “straw” layer. As we mentioned in the previous chapter, the “hump of pain” that most programmers have to overcome to master unit test automation means that the team may start out with only a few bricks. The fixtures that automate functional tests in the middle layer are easy to write if the system is designed with those tests in mind, so the sticks might pile up faster than the bricks. As teams master TDD and unit test automation, the bottom layer starts to grow. When they get traction, a team using TDD will quickly build out the brick foundation of the test pyramid.

See the bibliography for a link to Patrick Wilson-Welsh’s discussion of “flipping the test pyramid” right-side up.

The testing pyramid is a good place to start looking at how test automation can help an agile team. Programmers tend to focus on the bottom of the pyramid, and they need plenty of time and training to get over the “hump of pain” and get to the point where TDD is natural and quick. In traditional teams, testers usually have no choice but to automate tests at the GUI level. The whole-team approach used by agile teams means that testers pair with programmers and help them get better at writing tests, which in turn solidifies that brick foundation layer of the pyramid. Because tests drive development, the whole team is always designing for maximum testability, and the pyramid can grow to the right shape.

Programmers pair with testers to automate functional-level tests, filling out the middle layer. For example, a tester and customer may prepare a 400-row spreadsheet of test cases for a web services application. The programmer can help figure out a way to automate those tests. Different team members may have expertise in areas such as generating test data or using tools such as Excel macros, and all that knowledge spreads around the team. Working together, the team finds the best combinations of tools, test cases, and test data.

Involving the programmers in finding cost-effective ways to automate the top-level GUI tests has multiple benefits. These efforts may give programmers a better understanding of the system’s “big picture,” and testers can learn how to create more pliable, less straw-like GUI tests.

The more a team can work together and share knowledge, the stronger the team, the application, and the tests will become. The Big Bad Wolf won’t stand a chance. Let’s start by looking at what kind of tests we can automate and then at what we shouldn’t even try.

What Can We Automate?

Most types of testing you can think of benefit from automation. Manual unit tests don’t go far toward preventing regression failures, because performing a suite of manual tests before every check-in just isn’t practical. You can’t design code test-first through manual unit tests either. When programmers can’t run tests quickly at the touch of a button, they may not be motivated enough to run tests at all. We could manually test that different units of code work together correctly, but automated component tests are a much more effective safety net.

Manual exploratory testing is an effective way to find functional defects, but if we don’t have enough automated business-facing regression tests, we probably spend all of our time madly trying to keep up with manual regression testing. Let’s talk about all of the different kinds of testing that can be done well with automation.

To run automated tests, you need some kind of automated framework that allows programmers to check in code often, run tests on that code, and create deployable files. Let’s consider this first.

Continuous Integration, Builds, and Deploys

Any tedious or repetitive task involved in developing software is a candidate for automation. We’ve talked about the importance of an automated build process. You can’t build your automated test pyramid without this. Your team needs the immediate feedback from the unit-level tests to stay on track. Getting automated build emails listing every change checked in is a big help to testers because they know when a build is ready to test without having to bother the programmers.

See Chapter 7, “Technology-Facing Tests that Support the Team,” for examples of build automation tools.

Peril: Waiting for Tuesday’s Build

In a traditional environment, it is normal for testers to wait for a stable build, even if that means waiting until next Tuesday. In an agile environment, if testers don’t keep up with the developers, the stories get tested late in the game. If the developers don’t get the feedback, such as suggestions and bugs, the testers can lose credibility with the developers. Bugs won’t be discovered until the developers are already on another story and do not want to be interrupted to fix them until later.

Bugs pile up, and automation suffers because it can’t be completed. Velocity is affected because a story cannot be marked “done” until it is tested. This makes it harder to plan the next iteration. At the end of the release cycle, your story testing runs into the end game and you may not have a successful release. At the very least, you will have a stressful release.

An automated deployment process also speeds up testing and reduces errors. In fact, the day Janet was editing this chapter, she messed up the deployment because it was a manual process. It was pretty simple, but she was new to the project and moved the file to the wrong place. Getting an automated deployment process in place went on Janet’s list of things to get done right away. Lisa’s team implemented its continuous integration and build framework first thing, and found it fairly easy and quick to do, although it requires continual care and feeding. Other teams, especially those with large, complex systems, face much bigger hurdles.

We’ve talked with teams who had build times of two hours or more. This meant that a programmer would have to wait for two hours after checking in code to get validation that his check-in didn’t break any preexisting functionality. That is a long time to wait.

Most agile teams find an ongoing build longer than eight to ten minutes to be unworkable. Even 15 minutes is much too long to wait for feedback, because check-ins will start stacking up, and testers will wait a long time to get the latest, greatest build. Can you imagine how the developers working with a build that takes two hours feel as they approach the end of an iteration or release cycle? If they break any functionality, they’ll have to wait two more hours to learn whether or not they had fixed it.

Many times, long builds are the result of accessing the database or trying to test through the interface. Thousands of tests running against a large codebase can tax the resources of the machine running the build. Do some profiling of your tests and see where the bottleneck is. For example, if it is the database access that is causing most of the problems, try mocking out the real database and use an in-memory one instead. Configure the build process to distribute tests across several machines. See if different software could help manage resources better. Bring in experts from outside your team to help if needed.

The key to speeding up a continuous integration and build process is to take one small step at a time. Introduce changes one at a time so that you can measure each success separately and know you are on the right track. To start with, you may want to simply remove the most costly (in terms of time) tests to run nightly instead of on every build.

A fast-running continuous integration and build process gives the greatest ROI of any automation effort. It’s the first thing every team needs to automate. When it’s in place, the team has a way to get quick feedback from the automated tests. Next, we look at different types of tests that should be automated.

See the bibliography for links to build automation tools and books with more information about improving the build process.

Unit and Component Tests

We can’t overemphasize the importance of automating the unit tests. If your programmers are using TDD as a mechanism to write their tests, then they are not only creating a great regression suite, but they are using them to design high-quality, robust code. If your team is not automating unit tests, its chances of long-term success are slim. Make unit-level test automation and continuous integration your first priority.

Chapter 7, “Technology-Facing Tests that Support the Team,” goes into detail about some of the tools that can be used.

API or Web Services Testing

Testing an API or web services application is easiest using some form of automation. Janet has been on teams that have successfully used Ruby to read in a spreadsheet with all of the permutations and combinations of input variables and compare the outputs with the expected results stored in the spreadsheets. These data-driven tests are easy to write and maintain.

One customer of Janet’s used Ruby’s IRB (Interactive Ruby Shell) feature to test the web services for acceptance tests. The team was willing to share its scripts with the customer team, but the business testers preferred to watch to see what happened if inputs were changed on the fly. Running tests interactively in a semiautomated manner allowed that.

Testing behind the GUI

Testing behind the GUI is easier to automate than testing the GUI itself. Because the tests aren’t affected by changes to the presentation layer and work on more stable business logic code, they’re more stable. Tools for this type of testing typically provide for writing tests in a declarative format, using tables or spreadsheets. The fixtures that get the production code to operate on the test inputs and return the results can generally be written quickly. This is a prime area for writing business-facing tests, understandable to both customers and developers that drive development.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for specific tool examples.

Testing the GUI

Even a thin GUI with little or no business logic needs to be tested. The fast pace of agile development, delivering new functionality each iteration, mandates some automated regression tests at the GUI level for most projects. Tool selection is key for successful GUI automation. The automated scripts need to be flexible and easy to maintain. Janet has used Ruby and Watir very successfully when the framework was developed using good coding practices, just as if it were a production application. Time was put into developing the libraries so that there was not a lot of rework or duplication in the code, and changes needed could be made in one place. Making the code easy to maintain increased the ROI on these tests.

A point about testability here—make sure the programmers name their objects or assign IDs to them. If they rely on system-generated identifiers, then every time a new object is added to the page, the IDs will change, requiring changes to the tests.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for examples of GUI test frameworks.

Keep the tests to just the actual interface. Check things like making sure the buttons really work and do what they are supposed to. Don’t try to try to test business functionality. Other types of tests that can be automated easily are link checkers. There is no need for someone to manually go through every link on every page to make sure they hit the right page. Look for the low-hanging fruit, automate the things that are simple to automate first, and you’ll have more time for the bigger challenges.

Load Tests

Some types of testing can’t be done without automation. Manual load tests aren’t usually feasible or accurate, although we’ve all tried it at one time or another. Performance testing requires both monitoring tools and a way to drive actions in the system under test. You can’t generate a high-volume attack to verify whether a website can be hacked or can handle a large load without some tool framework.

See Chapter 11, “Critiquing the Product Using Technology-Facing Tests,” for examples of load test automation tools.

Comparisons

Visually checking an ASCII file output by a system process is much easier if you first parse the file and display it in a human-readable format. A script to compare output files to make sure no unintentional changes were made is a lot faster and more accurate than trying to compare them manually. File comparison tools abound, ranging from the free diff to proprietary tools such as WinDiff. Source code management tools, and IDEs have their own built-in comparison tools. These are essential items in every tester’s toolbox. Don’t forget about creating scripts for comparing database tables when doing testing for your data warehouse or data migration projects.

Read more about source code management tools and IDEs in Chapter 7, “Technology-Facing Tests that Support the Team.”

Repetitive Tasks

As we work with our customers to better understand the business and learn what’s valuable to them, we might see opportunities to automate some of their tasks. Lisa’s company needed to mail several forms with a cover letter to all of their clients. The programmers could not only generate the forms but could also concatenate them with the cover letter and greatly speed up the mailing effort. Lisa’s fellow tester, Mike Busse, wrote a spreadsheet macro to do complex calculations for allocating funds that the retirement plan administrators had been doing manually. A lot of manual checklists can be replaced with an automated script. Automation isn’t just for testing.

Data Creation or Setup

Another useful area for automation is data creation or setup. If you are constantly setting up your data, automate the process. Often, we need to repeat something multiple times to be able to recreate a bug. If that can be automated, you will be guaranteed to have the same results each time.

Lisa’s Story

Many of our test schemas, including the ones used for automated regression suites, use canonical data. This canonical or “seed” data was originally taken from production. Some tables in the database, such as lookup tables, don’t change, so they never need to be refreshed with a new copy. Other tables, such as those containing retirement plan, employee, and transaction information, need to start from Ground Zero whenever a regression suite runs.

Our database developer wrote a stored procedure to refresh each test schema from the “seed” schema. We testers may specify the tables we want refreshed in a special table called REFRESH_TABLE_LIST. We have an ant target for each test schema to run the stored procedure that refreshes the data. The automated builds use this target, but we use it ourselves whenever we want to clean up our test schema and start over.

Many of our regression tests create their own data on top of the “seed” data. Our Watir tests create all of the data they need and include logic that makes them rerunnable no matter what data is present. For example, the script that tests an employee requesting a loan from his or her retirement plan first cancels any existing loans so a new one can be taken out.

FitNesse tests that test the database layer also create their own data. We use a special schema where we have removed most constraints, so we don’t have to add every column of every table. The tests only add the data that’s pertinent to the functionality being tested. Each test tears down the data it created, so subsequent tests aren’t affected, and each test is independent and rerunnable.

—Lisa

Cleaning up test data is as important as generating it. Your data creation toolkit should include ways to tear down the test data so it doesn’t affect a different test or prevent rerunning the same test.

We’ve looked at major areas where automation is required or at least useful. Our opinion is that whenever you need to do a test or some testing-related activity, first decide whether it can be aided by automation. In some cases, automation won’t be appropriate. Let’s look at some of those.

What Shouldn’t We Automate?

Some testing needs human eyes, ears, and intelligence. Usability and exploratory testing are two that fall into that category. Other tests that may not justify the automation investment are one-off tests and those that will never fail.

Usability Testing

Real usability testing requires someone to actually use the software. Automation might be helpful in setting up scenarios to subsequently examine for usability. Observing users in action, debriefing them on their experiences, and judging the results is a job for a person who understands that usability aspects of software cannot be automated. Logging user actions is helpful for usability testing.

We discuss some logging and monitoring tools in Chapter 10, “Business-Facing Tests that Critique the Product.”

Janet’s Story

We had evaluated several GUI tools but decided to use Ruby with Watir. We kept our tests limited to GUI functions only. One of our tests was checking to make sure that correct validation messages were displaying on the screen. I was running the tests and happened to be watching the screen because I hadn’t seen this particular test that one of the other testers created. My eyes caught something weird, but the test passed, so I replayed it again. One of the programmers had added a “$” to the screen, and the error message was displayed offset because of it. The correct message was displayed, just not in the right place. In this instance, the value in watching the tests run was huge because we were preparing to release fairly soon, and we probably wouldn’t have caught that particular problem.

—Janet

It is possible to automate tests that make sure the GUI never changes, but you need to ask yourself whether it’s worth the cost. Do you really care that a button has changed positions by one pixel? Do the results justify the effort? We don’t think you should automate “look and feel” testing, because an automated script can only look for what you tell it to see. Automation would miss visual problems that would jump out at a human.

Exploratory Testing

Similarly, exploratory testing may be speeded up with scripts to create test data and jump through some setup steps, but it requires a skilled tester to design and execute the tests. One major goal of exploratory testing is to learn more about the product by doing, and then use that information to improve future development. Automated scripts won’t do that for you. However, as we’ve said before, you won’t have time for exploratory testing without a lot of other automated tests.

See Chapter 10, “Business-Facing Tests that Critique the Product,” for more on exploratory testing and tools that can facilitate it.

Tests that Will Never Fail

We’ve heard an argument that tests that will never fail don’t need to be automated. If a requirement is so obvious that there’s only one way to implement it, and no programmer will ever look at that code later without knowing exactly what it should do, the chances of someone introducing a defect in that code are next to nothing. Let’s say we have a form with address fields. Do we need an automated regression test to verify that the second street address line is not required? After we’ve verified it manually, how likely is it that someone will accidentally change it to a required field later? Even if someone did, it wouldn’t be a catastrophic event. Someone else would notice it and people could work around it easily until it was fixed.

Then again, a test for it would be easy to include. And programmer tricks such as copy/paste errors happen all the time. If you feel comfortable that one-time manual testing does the job and that the risk of future failures doesn’t justify automating regression tests, don’t automate them. If your decision turns out to be wrong, you’ll get another chance to automate them later. If you aren’t sure, and it’s not terribly difficult to automate, go for it.

If you’re testing a life-critical system, even a very small risk of a regression failure is too much. Use risk analysis to help decide what tests should be automated.

See Chapter 18, “Coding and Testing,” for more about risk analysis and how it relates to testing.

One-Off Tests

Most times, manually executing a one-off test is sufficient. If automating a test doesn’t have payoff, why do it? Sometimes automation is worth doing for a one-off test.

Lisa’s Story

We recently did a story to pop up a warning message dialog when posting a payroll, but the message should only come up during the first two weeks of January. Automating a test for this functionality would require some way to simulate that the current date was between January 1 and January 15. That’s not terribly hard to do, but the consequences of a failure were fairly trivial, and we had more critical stories to deliver that iteration. Automating that test at that time just didn’t have enough value to justify the cost, and the risk factor was low. We decided to test it manually.

There are other cases where doing a one-off test seems the most intuitive but automation is a better choice. We host sites for different business partners, and each one has unique content, look, and feel. Values in the database drive the correct behavior and content for each brand. Some of the data, such as fee schedules based on asset values and numbers of participants, are highly complex. It’s much easier and much more accurate to verify this data using FitNesse tests. We have a set of fixtures that let us specify keys for the partner “brand” that we want to test. We can easily plug in the appropriate expected results from the spreadsheets that the business development staff creates for each new partner. These tests aren’t part of our regression suite. They’re used one time only to validate the new brand.

—Lisa

Tedious tasks may be worth automating, even if you don’t do them often. Weigh the automation cost against the amount of valuable time eaten up by manually doing the test. If it’s easy to do manually, and automating wouldn’t be quick, just keep it manual.

What Might Be Hard to Automate?

When code isn’t written test-first, or at least with test automation in mind, it’s much harder to automate. Older systems tend to fall into this category, but no doubt plenty of new code with the same untestable characteristics is still being produced.

If you’re faced with working on existing code that doesn’t already have automated tests, you’re in for an uphill battle, but a winnable one. Legacy code may have I/O, database access, as well as business logic and presentation code intertwined. It may not be clear where to hook into the code to automate a test. How do you get started automating tests on such a system? You certainly can’t plan on automating everything below the GUI, because much of the logic is in the presentation layer.

There are at least a couple of different approaches that work well. The “hump of pain” we talked about in Chapter 13, “Why We Want to Automate Tests and What Holds Us Back,” is intimidating, but it can be overcome, and then test automation will become much easier. Michael Feathers’ Working Effectively With Legacy Code [2004] explains how to build a test harness around existing code bases and refactor them to accommodate automation. Even with legacy code, you can write tests to protect against introducing new problems. This approach can work even on systems that lack structure or aren’t object-oriented.

Lisa’s team decided on a different but equally effective approach. The team members started “strangling” the legacy code by writing all new features in a new test-friendly architecture. They’re gradually replacing all of the old code with code written test-first. When they do work on old code to fix bugs, or in the cases where the old code needs updating, they simply add unit tests for all of the code they change. A GUI smoke test suite covers the critical functions of the rest of the legacy system that has no unit tests.

Chapter 7, “Technology-Facing Tests that Support the Team,” goes into more detail about different agile approaches to legacy code.

As with any automation project, approach the hard-to-automate code one piece at a time, and address the highest risk areas first. Solve the testability problem and find a way to write unit-level tests. The effort will pay off.

Developing an Automation Strategy—Where Do We Start?

A simple, step-by-step approach sounds incompatible with an automation strategy, but in agile testing we try to understand the problem first. Deciding where and how to start with automation requires a bit of thought and discussion. As your team looks at testing challenges, you’ll need to consider where automation is appropriate. Before you start searching for a particular automation tool, you’ll want to identify your requirements.

You need to understand what problem you are trying to solve. What are you trying to automate? For example, if you have no test automation of any kind, and you start by buying an expensive commercial test tool thinking it will automate all your functional tests, you may be starting in the wrong place.

We suggest you start at the beginning. Look for your biggest gain. The biggest bang for the buck is definitely the unit tests that the programmers can do. Instead of starting at the top of the test pyramid, you may want to start at the bottom, making sure that the basics are in place. You also need to consider the different types of tests you need to automate, and when you’ll need to have tools ready to use.

In this section, we assume you have automated Quadrant 1 unit and component tests in place, and are looking to automate your business-facing tests in Quadrants 2 and 3, or your Quadrant 4 technology-facing tests that critique the product. We’ll help you design a good strategy for building your automation resources.

Think about the skills and experience on your team. Who needs the automation, and why? What goals are you trying to achieve? Understanding some of these issues may affect your choice of tools and what effort you expend. There is a section on evaluating tools at the end of this chapter.

Automation is scary, especially if you’re starting from scratch, so where do we begin?

Where Does It Hurt the Most?

To figure out where to focus your automation efforts next, ask your team, “What’s the greatest area of pain?” or, for some teams, “What’s the greatest area of boredom?” Can you even get code deployed in order to test it? Do team members feel confident about changing the code, or do they lack any safety net of automated tests? Maybe your team members are more advanced, have mastered TDD, and have a full suite of unit tests. But they don’t have a good framework for specifying business-facing tests, or can’t quite get a handle on automating them. Perhaps you do have some GUI tests, but they’re extremely slow and are costing a lot to maintain.

Peril: Trying to Test Everything Manually

If you’re spending all your time retesting features that you’ve tested before, not getting to new features, and needing to add more and more testing, you’re suffering from a severe lack of test automation. This peril means that testers don’t have time to participate in design and implementation discussions, regression bugs may creep in unnoticed, testing can’t keep up anymore with development, and testers get stuck in a rut. Developers aren’t getting involved in the business-facing testing, and testers don’t have time to figure out a better way to solve the testing problems.

Your team can fix this by developing an automation strategy, as we describe in this chapter. The team starts designing for testability and chooses and implements appropriate automation tools. Testers get an opportunity to develop their technical skills.

Wherever it hurts the most, that’s the place to start your automation efforts. For example, if your team is struggling to even deliver deployable code, you need to implement an automated build process. Nothing’s worse than twiddling your thumbs while you wait for some code to test.

But, if performance puts the existence of your organization in danger, performance testing has to be the top priority. It’s back to understanding what problem you are trying to solve. Risk analysis is your friend here.

Chapter 18, “Coding and Testing,” has more information on a simple approach to risk analysis.

Janet’s Story

I worked on a legacy system that was trying to address some quality issues as well as add new features for our main customer. There were no automated unit or functional tests for the existing application, but we needed to refactor the code to address the quality issues. The team members decided to tackle it one piece at a time. As they chose a chunk of functionality to refactor, the programmers wrote unit tests, made sure they passed, and then rewrote the code until the tests passed again. At the end of the refactoring, they had testable, well-written code and the tests to go with them. The testers wrote the higher-level functional tests at the same time. Within a year, most of the poor-quality legacy code had been rewritten, and the team had achieved good test coverage just by tackling one chunk at a time.

—Janet

Test automation won’t pay off unless other good development practices are in place. Continuous integration running a robust suite of unit tests is a first step toward automating other tests. Code that’s continually refactored for maintainability and good design will help increase the ROI on automation. Refactoring can’t happen without that good unit test coverage. These development practices also need to be applied to the automated functional test scripts.

Multi-Layered Approach

While we recommend mastering one tool at a time, don’t expect too much out of any one tool. Use the right tool for each need. The tool that works best for unit tests may or may not be appropriate to automate functional tests. GUI, load, performance, and security testing may each require a different tool or tools.

Mike Cohn’s test pyramid concept (see Figure 14-2) has helped our teams put their automation efforts where they do the most good. We want to maximize the tests that have the best ROI. If the system architecture is designed for testability, test automation will be less expensive, especially at the unit level.

Tests that go through the user interface usually have the lowest ROI, because they’re expensive to maintain, but we do need them. They make up the small tip of our pyramid We may choose to automate some of these tests, but the majority of GUI tests are defined in business terms and probably are best left as human interaction tests (i.e., manual tests).

The middle layer represents the functional tests that work directly with production code, without a GUI or other layer in between. While they’re not as inexpensive to automate as unit-level tests, and provide feedback a bit more slowly, the right tools allow them to have a good ROI. The fact that these tests can be written in a language the business experts understand adds to their value.

There are many different layers in the application that can be tested independently. In his book xUnit Test Patterns [2007], Gerard Meszaros refers to this as the Layer Test pattern. He cautions that when trying to test all of the layers of the application separately, we still have to verify that the layers are hooked up correctly, and this may require at least one test of the business logic through the presentation layer.

Lisa’s Story

As my team built our automation framework one step at a time, we gathered an arsenal of tools. After implementing a continuous build framework with Ant and CruiseControl, we mastered JUnit for unit testing. We knew that unit test automation is the quickest and cheapest way to automate, and provides the fastest feedback to the programmers.

Our legacy system had no automated tests, so we built a GUI regression test suite with Canoo WebTest. This provided good payback because the WebTest scripts were specified, not programmed. They were quick to write and easy to maintain.

After JUnit and WebTest were in place, we experimented with FitNesse and found it worked well for functional testing behind the GUI. We found automating with FitNesse to go relatively quickly. Although FitNesse tests are significantly more expensive to produce and maintain than unit tests, their value in driving development and promoting collaboration among customers, programmers, and testers kept the ROI high.

All of these tools were easy to learn, implement, and integrate with the build process, and provided continual feedback about our regression issues. They were important considerations when we were deciding on our test automation strategy.

—Lisa

When evaluating the payback of your automation efforts, consider less tangible aspects such as whether the tool promoted collaboration between the technical and customer teams. A primary reason to write tests is to help guide development. If the process of writing your automated acceptance tests results in a thorough understanding of business requirements, that’s plenty of payback, even if the tests never find a single regression bug later on.

Think about Test Design and Maintenance

Think about all of the manual test scripts you’ve written in your life. Don’t you just wish those all would have been automated? Wouldn’t your life have been a lot easier? We believe that all scripted tests should be automated. Let’s get started converting those manual scripted tests.

After you get started, it can be quite easy to automate tests. For example, when you have a working FitNesse fixture, adding more test cases requires little effort. This is great when you have a lot of different permutations to test. You’ll probably test more conditions than you would if all your testing was done manually. When Lisa’s team members rewrote their retirement plan’s loan system, they could test hundreds of different possibilities for loan payment processing via FitNesse tests. What happens when three loan payments are processed on the same day? If someone doesn’t make any payments for three months, and then sends in a large payment, is the interest calculated and applied correctly? It was easy to write automated tests to find out.

That’s a great advantage, but it has a down side. Now the team has dozens, or even hundreds, of test cases to maintain. What if the rules about calculating the amount of interest for a loan payment change a year from now? This could require updating every test. If your test tool doesn’t easily accommodate making changes to existing tests, your big suite of automated tests can turn into a headache.

End-to-end tests are particularly tricky to automate because they have the most potential to need maintenance as business rules change. How do we balance the need for automation with the cost?

Test Design

Remember to start with the thin slice or steel thread of the feature you’re testing. Approach automation just as programmers approach coding. Get one small unit of the steel thread working, and then move on to the next. After you’ve covered the whole thin slice, go back and flesh it out.

Chapter 8, “Business-Facing Tests that Support the Team,” explains more about thin slices.

Choose your test pattern thoughtfully. Automate all of the test cases you need, but no more, and automate them at the lowest level that you can. Limit the scope of each test case to one test condition or one business rule. Understand the purpose of the test. Avoid dependencies between tests, because they quickly increase complexity and maintenance expense.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for more information about effective test design.

Consider Options

As we’ve mentioned before, the lower the level at which you automate a test, the better the ROI. Push test automation as far down the pyramid as you can. If you have good coverage in your unit and code integration tests, you don’t need to automate as many functional tests. With solid coverage at the lower levels, it might be enough to do end-to-end tests manually to verify the system’s behavior. Use risk analysis to help you decide.

User Interface

The user interface does need to be tested. In some situations, test automation at the GUI level is critical. Perhaps your team is using third-party GUI controls, and you aren’t sure how they will behave. If your risk and ROI analysis supports a lot of automation at the GUI level, make the investment.

If you do automate at the higher levels, don’t go overboard and automate every possible path through the system. You don’t have to keep every automated test created during the development phase in the regression suite; consider tradeoffs of build time and the chance of finding defects. Focus your efforts on covering every important path through the code at the unit, code integration, and functional levels. You’ll get a much better payback.

Strike a Balance

Striking a balance isn’t an agile principle, it’s just common sense. You need a good-enough solution right now, but it doesn’t have to be perfect. Does the tool provide the results you need right now? Does it provide an adequate return on the resources needed to use it for automation? If so, go ahead and use it, and budget time later to look for alternatives. You can improve your automation framework over time. The most important factor is whether your automation tools fit your particular situation right now.

Don’t slide the other way, and think, “OK, we can generate a bunch of scripts with this record tool, get our immediate testing done, and refactor the scripts later to make them maintainable.” While you don’t need to keep searching for the perfectly ideal automation solution, you do need a solution that doesn’t add to your team’s technical debt. Find a balance between “It finds the bugs we need to know about and doesn’t cost too much to maintain” and “This is the most elegant and cool solution we can find.”

Choosing the Right Tools

It’s cool that we have so many tools available to help us solve our automation problems. Don’t go for more sophistication than you need. Lisa’s coworkers have found that a spreadsheet that retrieves data from the database and performs calculations independently of the system is a powerful tool, both for driving development and for verifying the application’s calculations.

We usually minimize test automation at the GUI layer, but there are situations where more GUI automation is appropriate. If the user makes a change at X, what else changes? Some problems only manifest themselves at the GUI level. Lisa tested a bug fix that addressed a back-end problem when retirement plan participants requested a distribution of money from their accounts. The change was surrounded by unit tests, but it was a GUI regression test that failed when the distribution form failed to pop up upon request. Nobody anticipated that a back-end change could affect the GUI, so they probably wouldn’t have bothered to test it manually. That’s why you need GUI regression tests, too.

We’ve talked about some disadvantages of record/playback tools, but they’re appropriate in the right situation. You may be using a record/playback tool for a good reason: Maybe your legacy code already has a suite of automated tests created in that tool, your team has a lot of expertise in the tool, or your management wants you to use it for whatever reason. You can use recorded scripts as a starting point, then break the scripts into modules, replace hard-coded data with parameters where appropriate, and assemble tests using the modules as building blocks. Even if you don’t have much programming experience, it’s not hard to identify the blocks of script that should be in a module. Login, for example, is an obvious choice.

Record/playback may also be appropriate for legacy systems that are designed in such a way that makes unit testing difficult and hand-scripting tests from scratch too costly. It’s possible to build a record and playback capability into the application, even a legacy application. With the right design, and the use of some human-readable format for the recorded interaction, it’s even possible to build playback tests before the code is built.

GUI Test Automation: From the Dark Ages to Successful Automation in an Agile Environment

Pierre Veragen, SQA Lead at iLevel by Weyerhaeuser, explains how his team used a tool that provided both record/playback and scripting capability productively in a waterfall environment, and then leveraged it when the company adopted agile development.

Back in our waterfall development days, in 2000, we started doing GUI test automation using a record-playback approach. We quickly accumulated tens of thousands of lines of recorded scripts that didn’t meet our testing needs. When I took over 18 months later, I quickly became convinced that the record-playback approach was for the dinosaurs.

When we had a chance to obtain a new test automation tool at the end of 2003, we carefully evaluated tools with these criteria in mind: record capability to help us understand the scripting language, and the ability to build an object-oriented library to cover most of our needs, including test reporting. At the time, TestPartner from CompuWare fulfilled all of our requirements.

We started using TestPartner on a highly complex, CAD-with-engineering application, built in Visual Basic 6, still using a waterfall process. Before we started automating tests, our releases were quickly followed by one or more patches. We focused our automation efforts toward checking the engineering calculations through the GUI, and later, the actual position of the CAD details. These tests included hundreds of thousands of individual verification points, which could never have been done by hand. Within a year, having added a solid set of manual tests of the user interaction, in addition to our automated tests, we were releasing robust software without the usual follow-up patches. We felt confident about our combination of manual and automated tests, which didn’t include a single line of recorded scripts.

In 2004, our group moved to Visual Basic .NET. I spent several months adapting our TestPartner library to activate .NET controls. In 2006, we adopted an Agile methodology. Building on lessons previously learned in the non-Agile world, we achieved astonishing results with test automation. By the end of 2006, team members were able to produce maintainable GUI test scripts and library components after just a few days of training. At the same time, the team embraced unit testing with NUnit and user acceptance tests with FitNesse.

As of this writing, issues are caught at all three levels of our automated testing: Unit, FitNesse, and GUI. The issues found by each of the three testing tiers are of a different nature. Because everything is automated and triggered automatically, issues are caught really fast, in true Agile fashion. Each part of our test automation is bringing value.

Some people feel resources would be better spent on architecture and design, so that GUI test automation isn’t needed. In our development group, each team made its own decision about whether to automate GUI tests.

In case you decide to use GUI test automation, here’s some advice: Stay away from recorded scripts, invest in maintainability, and minimize the required GUI testing with a good architecture of the application. It is my experience that investing in good GUI test automation practices will always pay off.

Pierre’s advice reflects well how good development practices, especially those followed in agile development projects, apply to automated test development as well as to production code development.

Built-In Record & Playback

Gerard Meszaros, agile coach and author of xUnit Test Patterns [2007], describes a situation where the simplest approach turned out to be record/playback. We’ve mentioned drawbacks to record/playback tools, but if you design your code to support them, they can be the best approach.

I was asked to help a team that was porting a “safety sensitive” application from OS2 to Windows. The business was very concerned about the amount of time it would take it to retest the ported system and the likelihood that the team would miss important bugs. The system was designed to only offer the user valid choices that would not compromise safety. They considered using a test recording tool to record tests on the old system and play them back on the new system, but there were no test recording tools available for both OS2 and Windows that could deal with windows drawn using ASCII characters. After reviewing the architecture of the system, we determined that writing xUnit tests would not be a cost-effective way to test the system because much of the business logic was embedded in the user interface logic, and refactoring the code to separate them would be too risky and time-consuming. Instead, we proposed building a Record & Playback test capability right into the system before we ported it.

Even though the rest of the project was milestone-driven, we developed the built-in test mechanism in a very agile way. Each screen required at least one new hook and sometimes several. We started with the most frequently used screens, adding the necessary hooks to record the user’s actions and the systems responses to them into an XML file. We also added the hooks to play back the XML and determine the test results. Initially, we focused our efforts on proving the concept by hooking only the screens we needed to record and then playing back a simple but realistic test. After everyone was convinced the approach would work, we prioritized the screens with respect to how much benefit it would provide. We implemented the hooks one by one until we could automate a significant portion of the tests. We also built an XSLT stylesheet that would format the XML in a Fit-like way, with green cells indicating acceptable results and red cells indicating a failed test step.

In the meantime, the client was identifying the test scenarios that needed test cases. As we finished enough screens to record a particular test, the client would “acceptance test” our hooks by recording and playing back (still on OS2) the test(s) that were waiting for those hooks. When all of the hooks were in place, we could go ahead and port the code, including the test hooks, from OS2 to Windows. After verifying successful playback on OS2, the client would move the XML test files over to Windows and run them against the ported version of the code. The client found this quite easy to do and was able to record a large number of tests in a relatively short period of time. Because the tests were recording actions and responses in business terms, the tests were fairly easy to understand. The client loved the capability, and still raves about how much effort it saved and how much more confidence it has in the product. “Not only did this save tens of man-years of testing effort, but it even uncovered hidden unknown bugs in the legacy system, which we had considered to be the gold standard.”

In Gerard’s story, the team worked together to retrofit testability onto a system that wasn’t designed for testability. They gave their customers a way to capture their test scenarios on one platform and play them back on both platforms to verify the successful port. This is a stellar example of the whole-team approach. When everyone on the team collaborates on a test automation solution, there’s a much better chance it’s going to succeed.

Some agile teams get value from commercial or open source test tools, while others prefer a completely customized approach. Many testers find value writing simple scripts in a scripting language such as Ruby, or a shell, to automate mundane but necessary tasks, generate test data, or drive other tools. Books such as Everyday Scripting with Ruby for Teams, Testers, and You give a roadmap for this approach. If you’re a tester without a strong programming background, we encourage you to pick up a book, find an online tutorial, or take a class on a scripting language, and see how easy it can be to write useful scripts.

See more examples of specific tools for business-facing tests in Chapter 9, “Toolkit for Business-Facing Tests that Support the Team.”

What we’re trying to tell you is that you can use many different tools. Look at the problem you are trying to solve and decide as a team the easiest and most effective way to solve it. Every so often, step back and take a look at the tools you’re using. Is everyone on the team happy with them? Are you missing problems because you don’t have the right tools? Budget time to explore new tools and see if they might fill gaps or replace a tool that isn’t paying off.

If your team is new to agile development, or working on a brand-new project, you might be faced with choosing tools and setting up test environments during the early iterations, when you also might be working on high-risk stories. Don’t expect to be able to deliver much business value if you’re still creating your test infrastructure. Plan in lots of time for evaluating tools, setting up build processes, and experimenting with different test approaches.

Applying Agile Principles to Test Automation

Every team, every project, and every organization has a unique situation with unique automation challenges. Each has its own culture, history, resources, business pressures, products, and experience. No matter what your team’s situation, you can use the agile principles and values discussed in Chapter 2 to help you find solutions. Concepts such as courage, feedback, simplicity, communication, continuous improvement, and responding to change aren’t just agile ideas—they’re qualities that are common to all successful teams.

Keep It Simple

The agile maxim of “do the simplest thing that could possibly work” applies to tests as well as code. Keep the test design simple, keep the scope minimal, and use the simplest tool that will do the job.

Simplicity is a core agile value for a good reason. The best place to start is the simplest approach you can think of. However, doing the simplest thing doesn’t mean doing the easiest thing. It involves really thinking about what you need now and taking baby steps to get there. By keeping things simple, if you do make a bad choice, you won’t go too far off track before realizing the error of your ways.

It’s easy to get involved in a task and slip away from the basics into some intriguing challenge. Weigh the ROI of every automation task before you do it. Automation is fun (when you get past the scary part of getting started). It’s tempting to try something difficult just because you can. Like all other aspects of testing in an agile development project, the only way to keep up is to do only the minimum required.

Use the simplest tool you can get away with. Remember the test pyramid. If a customer-facing test can be most easily automated at the unit level, do it there. Lisa sometimes writes test cases in FitNesse, only to learn the programmers can automate them much faster as JUnit tests. Conversely, sometimes the programmers use FitNesse for TDD instead of JUnit, because the code they’re writing lends itself to testing in one of the FitNesse fixture formats.

Iterative Feedback

Short iterations allow us to experiment with various automation approaches, evaluate results, and change course as quickly as needed. Commit to an automation effort, such as developing a test framework in-house or implementing an open source tool for at least a couple of iterations. After each iteration, look at what’s working and what’s not working. Think of ideas to overcome problems, and try those in the next iteration. If it’s not the right solution, try something else for a few iterations. Don’t get sucked into a quagmire where you’ve put so many resources into a tool, and have so many tests that use it, that you feel you can’t switch tools. Between the many open source and commercial tools, plus programmers’ ability to write home-grown test tools, there’s no reason to settle for less than the optimum tool.

Lisa’s Story

One of my early XP teams struggled to find a good way to automate customer-facing acceptance tests for a Java-based web application. This was back when there were far fewer tool options for agile teams. First, we tried an open source tool that simulated a browser, but it lacked the features we required. It just wasn’t quite robust enough. We discussed this at the next retrospective.

We decided to try using the unit testing tool for testing behind the GUI for the next two iterations. By committing to two iterations, we felt we were giving ourselves enough time to give the tool a good try, but not so much time that we would have too much invested if it weren’t the right solution. The customers found the unit tests hard to read, and there was logic in the GUI we couldn’t test with this tool.

After another discussion during our retrospective, we then committed to two iterations of using a vendor GUI test tool I had used extensively on previous projects. The Java programmers found it slow going because the tool used a proprietary scripting language, but it worked well enough to do the minimum automation needed. After two iterations, we decided that it wasn’t ideal, but at the time there weren’t a lot of other options, and it was the best one we had.

In hindsight, we should have kept looking for a better option. Perhaps we could have developed our own test harness. We were able to automate about 60% of the regression tests above the unit level using the vendor tool, which seemed great at the time. If we had pushed ourselves a little more, we might have done a lot better.

—Lisa

Use iterations to your advantage. They facilitate a step-wise approach. If your idea’s a dud, you’ll know quickly and have a chance to try a different one. Don’t be afraid to keep looking, but don’t keep looking for the perfect solution if one you try performs adequately.

Whole-Team Approach

Agile development can’t work without automation. Fortunately, the whole-team approach, which we explored in Chapter 1, means that a wider range of skills and resources are available to find and implement a useful automation strategy. Attacking the problem as a team means it’s more likely that code will be designed for testability. Programmers, testers, and other team members will collaborate to automate tests, bringing multiple viewpoints and skill sets to the effort.

The whole-team approach helps overcome the fear barrier. Automation tasks can be overwhelming to start with. Knowing there are other people with different skills and experience to help gives us courage. Being able to ask for and receive help gives us confidence that we can achieve adequate coverage with our automated tests.

Lisa’s Story

My current team made a commitment to automating regression tests at all levels where it made sense. Here are some examples of where I’ve asked for help to succeed with automation.

Early on, when we had no automated tests at all and the developers were trying to master test-driven development, we settled on Canoo WebTest for the GUI smoke tests. I needed a bit of help understanding how to configure WebTest to run in our environment, and I needed a lot of help to run the tests from the automated build process. I asked our system administrator (who was also one of the programmers) to help. We quickly got a suite of tests running in the build.

Later, I really wanted to try FitNesse for functional testing behind the GUI. I had to be patient while the programmers were still getting traction with the automated unit tests. The team agreed to try the tool, but it was hard to find time to start using it. I picked a story that seemed suited to FitNesse tests, and asked the programmer working on the story if I could pair with him to try some FitNesse tests. He agreed, and we got some tests automated in FitNesse. The programmer found it easy and worthwhile, and gave a good report to the rest of the team.

After that, it wasn’t hard to approach each programmer, suggest writing FitNesse tests for the story he was working on, and let him see the results. The FitNesse tests found test cases the programmer hadn’t thought of, and they saw the benefit right away. When everyone on the team had some experience with the tool, they were not only happy to automate the tests, but started designing code in a way that would make writing FitNesse fixtures easier.

When our Ruby expert, who designed most of our Watir test suite, left the company, I was quite concerned about maintaining our huge suite of tests as well as being able to code new ones. My Ruby expertise was not as good as his (plus, we were down to just one tester, so time was an issue). Every programmer on the team went out, bought a book on Ruby, and helped when I had problems updating scripts to work when the code changed. One programmer even wrote a new script to test a new story when I didn’t have time for that task. When we hired a new tester, he and I were able to handle the care and feeding of the Watir scripts, so the programmers no longer needed to take on those tasks.

I know I can ask teammates for help with automation issues, and the entire team sees automation as a priority, so the programmers always think about testability when designing the code. This is an example of the whole-team approach at work.

—Lisa

Specialized technology-facing tests such as security or load testing might require bringing in experts from outside the team. Some companies have specialist teams that are available as shared resources to product teams. Even while taking advantage of these resources, agile teams should still take responsibility for making sure all types of testing are done. They may also be surprised to find that team members may have the skills needed if they take a creative approach.

Chapter 11, “Critiquing the Product Using Technology-Facing Tests,” talks about technology-facing tests such as these and different approaches to handling them.

Some organizations have independent test teams that do post-development testing. They may be testing to ensure the software integrates with other systems, or conducting other specialized testing such as large-scale performance testing. Development teams should work closely with these other teams, using feedback from all testing efforts to improve code design and facilitate automation.

Taking the Time to Do It Right

Solving problems and implementing good solutions takes time. We must help our management understand that without enough time to do things the right way, our technical debt will grow, and our velocity will slow. Implementing solutions the “right” way takes time up front but will save time in the long term. Consider the time it takes for brainstorming ideas, solutions, formal training, and for on-the-job learning.

Your organization’s management is understandably interested in producing results as quickly as possible. If management is reluctant to give the team time to implement automation, explain the trade-offs clearly. Delivering some features in the short term without automated regression tests to make sure they keep working will have a big cost down the line. As your team accumulates technical debt, you’ll be less able to deliver the business value management needs. Work toward a compromise. For example, cut the scope of a feature but keep the essential value, and use automation to deliver and maintain a better product.

We always have deadlines, and we always feel pressed for time. The temptation to just go back to doing things the way we always have, like executing regression tests manually and hoping for the best, is always there even though we know that doesn’t work. There is never enough time to go back and fix things. During your next planning meeting, budget time to make meaningful progress on your automation efforts.

Lisa’s Story

Our team focuses on taking time for good design, a strong set of automated tests, and ample time for exploratory testing. Quality, not speed, has always been our goal. Our production problems cost a lot to fix, so the whole company is on board to take the time to prevent them. Sometimes we don’t pick the right design, and we aren’t afraid to rip it out and replace it when we realize it.

Naturally there are business tradeoffs, and the business decides whether to proceed with known risks. We work to explain all of the risks clearly and give examples of potential scenarios.

Here are a couple of recent examples of taking the time to do things right. We started a theme to make major changes to the account statements for the retirement plans. One of the programmers, Vince Palumbo, took on the task of collecting additional data to be used for the statements. He decided to write robust unit tests for the data collection functionality, even though this meant the story would have to continue on to the next iteration. Writing the unit tests took a great deal of time and effort, and even with the tests, the code was extremely complex and difficult to do. A couple of iterations later, another programmer, Nanda Lankalapalli, picked up another story related to the data collection and was pleasantly surprised to find new unit tests. He was able to make his changes quickly, and the testing effort was greatly reduced because the unit tests were in place.

Later, we found we had missed an edge case where some calculations for the change in account value were incorrect. The combination of automated unit tests and a great deal of exploratory testing were not enough to catch all of the scenarios. Still, having the tests meant Vince could write his corrected code test-first and feel more confident that the code was now correct.

Another recent example concerned processing of incoming checks. The business wanted to shorten the two-step process to one step, which meant the money would be invested in the retirement plan accounts two days earlier than was then possible. The existing process was all written in legacy code, without unit tests. We discussed whether to rewrite the processing in the new architecture. Our product owner was concerned about the amount of time this might take. We felt it would take just as long to change the existing code as to completely rewrite it, because the old code was difficult to understand and had no unit tests at all. We decided on the rewrite, which not only reduced the risk of problems in this critical functionality but also gave us the opportunity to provide a couple of extra features at little extra cost. So far, this strategy has proven worthwhile.

—Lisa

Allow yourself to succeed. Work at a sustainable pace. Take the time to refactor as you go or you’ll end up with a mess eventually. As testers, we always have many different tasks to do. If you’re learning a new tool or trying to automate new tests, don’t multitask. Find a big block of time and focus. This is hard, but switching gears constantly is harder.

If business stakeholders are impatient for your team to “just get it done,” analyze the problem with them. What are the risks? How much will a production problem cost? What are the benefits of releasing a quick hack? How much technical debt will it add? What’s the long-term return on investment of a solid design supported with automated tests? How will each approach affect company profitability and customer satisfaction? What about the intangible costs, such as the effect that doing poor-quality work has on team morale? Sometimes the business will be right, but we’re betting that you’ll usually find that up-front investment pays off.

Learn by Doing

Everyone learns in different ways, but when you’ve decided how you’re going to automate a test, jump in and start doing it. In Everyday Scripting with Ruby for Teams, Testers, and You [2007], Brian Marick advises to learn to program by writing a program. Make mistakes! The more problems you have, the more you’ll learn. Getting someone to pair with you will help speed up learning, even if neither one of you is familiar with the tool or the language.

If you don’t have anyone to pair with, talk to the “rubber ducky”: Imagine you’re describing the problem to a coworker. The process of explaining can often make the cause of the problem jump into view. Simply reading a test aloud to yourself can help you find the weaknesses in it.

Apply Agile Coding Practices to Tests

Tests are just as valuable as production code. In fact, production code isn’t much without tests to support it. Treat your tests the same way you treat all code. Keep it in the same source code control tool as your production code. You should always be able to identify the versions of test scripts that go with a particular version of code.

Pairing, refactoring, simple design, modular and object-oriented design, good standards, keeping tests as independent as possible—all of the qualities of good code are also qualities of good automated tests. Agile development is sometimes perceived by the uninformed to be chaotic or lax, when in fact it is highly disciplined. Undertake your automation tasks with the greatest discipline, proceeding in small steps, checking in each step that succeeds. If you’re programming automated scripts, write them test-first, just as any agile programmer would write production code. Keep simplicity in mind, though. Don’t write fancy test scripts with lots of logic unless there’s a good ROI. Those tests need testing and cost more to maintain. Specify tests when you can instead of coding them, and always go with the simplest approach possible.

We can’t emphasize it enough: Test automation is a team effort. The varying experience, skills, and perspectives of different team members can work together to come up with the best approach to automation. Innovate—be creative. Do what works for your unique situation, no matter what the “common wisdom” says.

Automation tools are just one piece of the puzzle. Test environments and test data are essential components. Let’s look at test data next.

Supplying Data for Tests

No matter what tool we use to automate tests, the tests need data to process. Ideally, they need realistic data that matches production data. However, production databases usually contain lots and lots of data, and they can be highly complex. Also, database access slows down tests exponentially. Like so much of agile testing, it’s a balancing act.

Data Generation Tools

As we write this book, there are several cool tools available to generate test data for all kinds of input fields and boundary conditions. Open source and commercial tools such as Data Generator, databene benerator, testgen, Datatect, and Turbo Data are available to generate flat files or generate data directly to database tables. These tools can generate huge varieties of different types of data, such as names and addresses.

It’s also fairly easy to generate test data with a home-grown script, using a scripting language such as Ruby or Python, a tool such a Fit or FitNesse, or a shell script.

Lisa’s Story

Our Watir scripts create randomized test data inputs, both to ensure they are rerunnable (they’re unlikely to create an employee with the same SSN twice), and to provide a variety of data and scenarios. The script that creates new retirement plans produces plans with about 200 different combinations of options. The script that tests taking out a loan randomly generates the frequency, reason, and term of the loan, and verifies that the expected payment is correct.

We have utility scripts to create comma-separated files for testing uploads. For example, there are several places in the system that upload census files with new employee information. If I need a test file with 1,000 new employees with random investment allocations to a retirement plan, I can simply run the script and specify the number of employees, the mutual funds they’re investing in, and the file name. Each record will have a randomly generated Social Security Number, name, address, beneficiaries, salary deferral amounts, and investment fund allocations. Here’s a snippet of the code to generate the investment calculations.

# 33% of the time maximize the number of funds chosen, 33% of the time

# select a single fund, and 33% of the time select from 2-4 funds

fund_hash = case rand(3)

when 0: a.get_random_allocations(@fund_list.clone)

when 1: a.get_random_allocations(@fund_list.clone, 1)

when 2: a.min_percent = 8;

a.get_random_allocations(@fund_list.clone, rand(3) + 2)

end

emp['fund_allocations'] = fund_hash_to_string(fund_hash)

Scripts like these have dual uses, both as regression tests that cover a lot of different scenarios and exploratory test tools that create test data and build test scenarios. They aren’t hard to learn to write (see the section “Learning by Doing” earlier in this chapter).

—Lisa

Scripts and tools to generate test data don’t have to be complex. For example, PerlClip simply generates text into the Windows clipboard so it can be pasted in where needed. Any solution that removes enough tedium to let you discover potential issues about the application is worth trying. “The simplest thing that could possibly work” definitely applies to creating data for tests. You want to keep your tests as simple and fast as possible.

Avoid Database Access

Your first choice for testing should try to have tests that can run completely in-memory. They will still need to set up and tear down test data, but the data won’t store in a database. Each test is independent and runs as quickly as any test could. Database access means I/O and disks are inherently slow. Every read to the database slows down your test run. If your goal is to give fast feedback to the team, then you want your tests to run as quickly as possible. A fake object such as an in-memory database lets the test do what it needs to do and still give instant feedback.

Lisa’s Story

One of our build processes runs only unit-level tests, and we try to keep its runtime less than eight minutes, for optimum feedback. The tests substitute fake objects for the real database in most cases. Tests that are actually testing the database layer, such as persisting data to the database, use a small schema with canonical data originally copied from the production database. The data is realistic, but the small amount makes access faster.

At the functional test level, our FitNesse test fixtures build data in-memory wherever possible. These tests run quickly, and the results appear almost instantaneously. When we need to test the database layer, or if we need to test legacy code that’s not accessible independently of the database layer, we usually write FitNesse tests that set up and tear down their own data using a home-grown data fixture. These tests are necessary, but they run slowly and are expensive to maintain, so we keep them to the absolute minimum needed to give us confidence. We want our build that runs business-facing tests to provide feedback within a couple of hours in order to keep us productive.

—Lisa

Comprehensive explanations and examples of various types of test doubles can be found in xUnit Test Patterns. See the bibliography for more information on that and tools for working with test stubs and with mock and fake objects.

Because it’s so difficult to get traction on test automation, it would be easy to say “OK, we’ve got some tests, and they do take hours to run, but it’s better than no tests.” Database access is a major contributor to slow tests. Keep taking small steps to fake the database where you can, and test as much logic as possible without involving the database. If this is difficult, reevaluate your system architecture and see if it can be organized better for testing.

Tools such as DbFit and NdbUnit can simplify database testing and enable test-driven database development; see the bibliography for more resources.

If you’re testing business logic, algorithms, or calculations in code, you’re interested in the behavior of the code itself given certain inputs; you don’t care where the data comes from as long as it accurately represents real data. If this is the case, build test data that is part of the test and can be accessed in memory, and let the production code operate from that. Simulate database access and objects, and focus on the purpose of the test. Not only will the tests run faster, but they’ll be easier to write and maintain.

When generating data for a test, use values that reflect the intent of the test, where possible. Unless you’re completely confident that each test is independent, generate unique test values for each test. For example, use timestamps as part of the field values. Unique data is another safety net to keep tests from infecting each other with stray data. When you need large amounts of data, try generating the data randomly, but always clean it up at the end of the test so that it doesn’t bleed into the next test. We recognize that sometimes you need to test very specific types of data. In these cases, randomly generated data would defeat the purpose of the test. But you may be able to use enough randomization to ensure that each test has unique inputs.

When Database Access Is Unavoidable or Even Desirable

If the system under test relies heavily on the database, this naturally has to be tested. If the code you’re testing reads from and/or writes to the database, at some point you need to test that, and you’ll probably want at least some regression tests that verify the database layer of code.

Setup/Teardown Data for Each Test

Our preferred approach is to have every test add the data it needs to a test schema, operate on the data, verify the results in the database, and then delete all of that test data so the test can be rerun without impacting other subsequent tests. This supports the idea that tests are independent of each other.

Lisa’s Story

We use a generic data fixture that lets the person writing the test specify the database table, columns, and values for the columns in order to add data. Another generic data lookup fixture lets us enter a table name and SQL where clause to verify the actual persisted data. We can also use the generic data fixture to delete data using the table name and a key value. Figure 14-3 shows an example of a table that uses a data fixture to build test data in the database. It populates the table “all fund” with the specified columns and values. It’s easy for those of us writing test cases to populate the tables with all of the data we need.

Figure 14-3 Example of a table using a data fixture to build test data in the database

Note that the schemas we use for these tests have most of their constraints removed, so we only have to populate the tables and columns pertinent to the functionality being tested. This makes maintenance a little easier, too. The downside is that the test is a bit less realistic, but tests using other tools verify the functionality with a realistic environment.

The downside to creating test data this way is that whenever a change is made in the database, such as a new column with a required value, all of the data fixture tables in the tests that populate that table will have to be changed. These tests can be burdensome to write and maintain, so we only use them when absolutely needed. We try to design the tests to keep maintenance costs down. For example, the data fixture in Figure 14-3 is in an “include” library and can be included into the tests that need it. Let’s say we add a new column, “fund_category.” We only need to add it to this “include” table, rather than in 20 different tests that use it.

—Lisa

Canonical Data

Another alternative is having test schemas that can quickly be refreshed with data from a canonical or seed database. The idea is that this seed data is a representative sample of real production data. Because it’s a small amount of data, it can be quickly rebuilt each time a suite of regression tests needs to be run.

This approach also increases the time it takes to run tests, but it’s just a few minutes at the start of the regression suite rather than taking time out of each individual test. The tests will still be slower than tests that don’t access the database, but they’ll be faster than tests that have to laboriously populate every column in every table.

Canonical data has many uses. Testers and programmers can have their own test schema to refresh at will. They can conduct both manual and automated tests without stepping on anyone else’s testing. If the data is carefully chosen, the data will be more realistic than the limited amount of data each test can build for itself.

Of course, as with practically everything, there’s a downside. Canonical data can be a pain to keep up. When you need new test scenarios, you have to identify production data that will work, or make up the data you need and add it to the seed schema. You have to scrub the data, mask real peoples’ identifying characteristics, making it innocuous for security reasons. Every time you add a table or column to the production database, you must update your test schemas accordingly. You might have to roll date-sensitive data forward every year, or do other large-scale maintenance. You have to carefully select which tables should be refreshed and which tables don’t need refreshing, such as lookup tables. If you have to add data to increase test coverage, the refresh will take longer to do, increasing the time of the build process that triggers it. As we’ve been emphasizing, it’s important that your automated builds provide feedback in a timely manner, so longer and longer database refreshes lengthen your feedback cycle. You also lose the test independence with canonical data, so if one test fails, others may follow suit.

Lisa’s team members run their GUI test suites and some of their functional regression tests against schemas refreshed each run with canonical data. On rare occasions, tests fail unexpectedly because of an erroneous update to the seed data. Deciding whether to “roll” data forward, so that, for example, 2008’s rows become 2009’s rows, gets to be a headache. So far, the ROI on using canonical data has been acceptable for the team. Janet’s current team also uses seed data for its “middle layer” testing on local builds. It works well for fast feedback during the development cycle. However, the test environment and the staging environments use a migrated copy of production data. The downside is that the regression tests can only be run on local copies of the build. The risk is low because they practice “build once, deploy to many.”

Production-Like Data

The ability to test a system that is as much like production as possible is essential to most software development teams. However, running a suite of automated regression tests against a copy of a production database would probably run too slowly to be useful feedback. Besides, you couldn’t really depend on any data remaining stable as you bring over new copies to stay up-to-date. Generally, when you’re talking about functional or end-to-end testing, a clone of the production database is most useful for manual exploratory testing.

Stress, performance, and load testing, which are automation–intensive, need an environment that closely simulates production in order to provide results that can translate to actual operations. Usability, security, and reliability are other examples of testing that needs a production-like system, although they may not involve much automation.

There is always a trade-off; your production database might be huge, so it is expensive and slow, but it provides the most accurate test data available. If your organization can afford hardware and software to store multiple copies of production data for testing purposes, this is ideal. Small companies may have resource constraints that might limit the amount of data that can be stored in test and staging environments. In this case, you’ll need to decide how much test data you can support and plan how to copy enough relevant data to make the test representative of what’s used in “real life.” Or you may consider making the investment in hardware, which is getting less expensive every day, to support a real production style environment. Otherwise, your test results might be misleading. As we mentioned with the canonical data, you may need to scrub the data before using it.

Data Migration

Data migration needs to be tested against a real database. The database upgrade scripts need to be run against real data and against the last known release of the database schema.

Testing a Database Migration

Paul Rogers, an automation test architect, tells this story of testing an eye-opening database migration [2008]:

Just yesterday, I ran a Rails migration against my test database. The developers had written it, tested it, and checked it using their development databases. My test database was probably 20,000 times larger. The migration for them took seconds. For me, well, I stopped it after three hours, at probably 10% complete. The programmers needed to redo their migration strategy.

I doubt this would have shown up on an in-memory database, so for me, a real database in this instance was definitely the right choice. In fact, this is likely to feed into things we need to consider before releasing, such as how long does a deployment take, or how long does the database update take. We can then use this to estimate how much down time we will need for the actual upgrade.

This is another example of how we must strike a balance between tests that deliver quick feedback and tests that realistically reflect events that might occur in production.

Understand Your Needs

If you understand the purpose of your tests, you can better evaluate your needs. For example, if you don’t need to test stored procedures or SQL queries directly for speed, consider tools such as in-memory databases, which work just like real databases but greatly speed up your tests. When you need to simulate the actual production environment, make a copy of the entire production database, if necessary. Quick feedback is the goal, so balance testing realistic scenarios with finding defects as efficiently as possible.

Evaluating Automation Tools

The first step in choosing an automation tool is to make a list of everything the tool needs to do for you. Let’s consider how you can decide on your test tool requirements.

Identifying Requirements for Your Automation Tool

After deciding on the next automation challenge to tackle, think about your tool needs. What tools do you already have? If you need additional ones, you probably want something that integrates well with your existing testing and development infrastructure. Do you need a tool to easily integrate into the continuous build process? Will your existing hardware support the automation you need to do? Setting up a second build process to run functional tests may require additional machinery.

Who’s going to use the test tool you’re hoping to implement? Will non-programmers be writing test cases? Do your programmers want a tool they feel comfortable with as well? Do you have distributed team members who need to collaborate?

Who will be automating and maintaining the tests? The skills already on your team are important. How much time do you have to get a tool installed and learn how to use it? If your application is written in Java, a tool that uses Java for scripting may be the most appropriate. Do team members have experience with particular tools? Is there a separate test team with expertise in a certain tool? If you’re starting the transition to agile development and you already have a team of test automators, it may make sense to leverage their expertise and keep using the tools they know.

Your tool requirements are dependent on your development environment. If you’re testing a web application, and the tool you choose doesn’t support SSL or AJAX, you may have a problem. Not every test tool can test web services applications. Embedded system testing can need different tools again. The case study in Chapter 12, “Summary of Testing Quadrants,” shows one way to use Ruby to test an embedded application.

Of course, the type of testing you’re automating is key. Security testing probably needs highly specialized tools. There are many existing open source and vendor tools for performance, so the job of selecting one isn’t overwhelming. As you master one challenge, you’ll be better prepared for the next. It took Lisa’s team a couple of years to develop robust regression test suites at the unit, integration, and functional levels. Performance testing was their next area of pain. Lessons learned from the earlier automation efforts helped them do a better job of identifying requirements for a test tool, such as ease of reporting results, compatibility with existing frameworks, and scripting language.

Write a checklist that captures all your tool requirements. Some of them might conflict with or contradict each other—“The tool needs to be easy enough so that customers can specify tests” or “The tests should be easy to automate.” Write them down so you can find the right balance. Then start doing your research.

One Tool at a Time

You’re going to need different tools to serve different purposes. Implementing new tools and learning the best way to use them can get overwhelming pretty quickly. Try one tool at a time, addressing your greatest area of pain. Give it enough time for a fair trial and evaluate the results. If it’s working for you, master that tool before you go on to the next area of pain and the next tool. Multitasking might work for some situations, but new technology demands full attention.

When you’ve settled on a tool to address a particular need, take a step back and see what else you need. What’s the next automation challenge facing your team? Will the tool you just selected for another purpose work for that need, too, or do you need to start a new selection process?

If you’ve decided to look outside your own organization for tools, the first step is to find time to try some out. Start with some basic research: Internet searches, articles and other publications about tools, and mailing lists are good places to get ideas. Compile a list of tools to consider. If your team uses a wiki or online forum tool, post information about tools and start a discussion about pros and cons.

The bibliography contains websites that help with tool searches and evaluation.

Budget time for evaluating tools. Some teams have an “engineering sprint” or “refactoring iteration” every few months where, rather than delivering stories prioritized by the business, they get to work on reducing technical debt, upgrading tool versions, and trying out new tools. If your team doesn’t have these yet, make a case to your management to get them. Reducing your technical debt and establishing a good testing infrastructure will improve your velocity in the future and free time for exploratory testing. If you never have time to make code easier to maintain or upgrade tools, technical debt will drag down your velocity until it comes to a halt.

When you have a list of tools that may meet your requirements, narrow the possibilities down to one or two, learn how to use each one well enough to try it, and do a spike: Try a simple but representative scenario that you can throw away. Evaluate the results against the requirements. Use retrospectives to consider pros and cons.

What resources do you need to implement and use the tool? What impact will the tool have on the team’s productivity and velocity? What risks does it pose? What will it allow you to do in the long term that you can’t do now?

Pick your top candidate and commit to trying it for some period of time—long enough to get some competency with it. Make sure you try all your mission-critical functionality. For example, if your application uses a lot of Ajax, make sure you can automate tests using the tool. In retrospectives, look at what worked and what didn’t. Be open to the idea that it might not be right and that you have to throw it out and start over. Don’t feel you have to keep on with the tool because you have so much invested in it already.

We all know that there’s no “silver bullet” that can solve all your automation problems. Lower your expectations and open your mind. Creative solutions rely on art as much as science.

Lisa’s Story

When conducting the performance test tool search, we turned to an agile testing mailing list for suggestions. Many people offered their experiences, and some even offered to help learning and implementing a tool. We searched for a tool that used Java for scripting, had a minimal learning curve, and presented results in a useful graphical format. We listed tools and their pros and cons on the team wiki. We budgeted time for trial runs. Lisa’s coworker, Mike Busse, tried the top two candidates and showed highlights to the rest of the team. A tool was chosen by team consensus and has proven to be a good fit.

—Lisa

Chapter 11, “Critiquing the Product Using Technology-Facing Tests, shows an example of the results produced by the performance test tool chosen, JMeter.

Choosing Tools

We’re lucky to have an already vast range and ever-growing set of tools to choose from: home-grown, open source, vendor tools, or a combination of any, are all viable alternatives. With so many choices, the trick is knowing where to look and finding time to try tools out to see if they fit your requirements. Because we can’t predict the future, it may be hard to judge the ROI of each potential solution, but an iterative approach to evaluating them helps get to the right one.

Should You Grow Your Own?

Does your application present unique testing challenges, such as embedded software or integration with outside systems? Do team members have the skills, time, and inclination to write their own test framework or build one on top of an existing open source tool? If so, home-grown test tools may be the best fit.

A happy result (or perhaps a major success factor) of agile development is that many programmers are “test infected.” Today’s development tools and languages make automation frameworks easier to build. Ruby, Groovy, Rails, and many languages and frameworks lend themselves to automation. Existing open source tools such as Fit and HtmlUnit can be leveraged, with custom frameworks built on top of them.

Home-grown tools have many advantages. They’re definitely programmer-friendly. If your team is writing its own automation frameworks, they’ll be precisely customized to the needs of your development and customer teams, and integrated with your existing build process and other infrastructure—and you can make them as easy to execute and interpret results as you need.

Home-grown doesn’t mean free, of course. A small team may not have the bandwidth to write and support tools as well as develop production code. A large organization with unique requirements may be able to put together a team of automation specialists who can collaborate with testers, customers, programmers, and others. If your needs are so unique that no existing tool supports them, home-grown may be your only option.

Open Source Tools

Many teams who wrote their own tools have generously made them available to the open source community. Because these tools were written by test-infected programmers whose needs weren’t met by vendor tools, they are usually lightweight and appropriate for agile development. Many of these tools are developed test-first, and you can download the test suite along with the source code, making customization easier and safer. These tools have a broad appeal, with features useful to both programmers and testers. The price is right, although it’s important to remember that purchase price is only a fraction of any tool’s cost.

Not all open source tools are well documented, and training can be an issue. However, we see seminars and tutorials on using these tools at many conferences and user group meetings. Some open source tools have excellent user manuals and even have online tutorials and scheduled classes available.

If you’re considering an open source solution, look for an active developer and user community. Is there a mailing list with lots of bandwidth? Are new features released often? Is there a way to report bugs, and does anyone fix them? Some of these tools have better support and faster response on bugs than vendor tools. Why? The people writing them are also using them, and they need those features to test their own products.

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” for more about open source test automation tools.

Vendor Tools

Commercial tools are perceived as a safe bet. It’s hard to criticize someone for selecting a well-known tool that’s been around for years. They’re likely to come with manuals, support, and training. For testers or other users who lack a technical background, the initial ramp-up might be faster. Some are quite robust and feature-rich. Your company may already own one and have a team of specialists who know how to use it.

Although they are changing with the times, vendor tools are historically programmer-unfriendly. They tend to use proprietary scripting languages that programmers don’t want to spend time learning. They also tend to be heavyweight. The test scripts may be brittle, easily broken by minor changes to the application, and expensive to maintain. Most of these tools are recording scripts for subsequent playback. Record/playback scripts are notoriously costly from a maintenance perspective.

Elisabeth Hendrickson [2008] points out that specialized tools such as these may create a need for test automation specialists. Silos such as these can work against agile teams. We need tools that facilitate test-first, rather than test-last development. Test tools shouldn’t stand in the way of change.

See the bibliography for a full discussion by Elisabeth Hendrickson on this subject.

If you have people already expert in a vendor tool, and a use for a tool that might be used only by a subset of the development team or a team separate from development, a vendor tool could make lots of sense. Lisa’s first two XP teams used a vendor tool with some degree of success.

As of this writing, better functional test tools and IDEs are emerging. These facilitate test maintenance tasks with features such as global search/replace. Twist is an example of a tool implemented as a collection of plug-ins to the Eclipse IDE, so it can take advantage of powerful editing and refactoring features.

Agile-Friendly Tools

Elisabeth Hendrickson [2008] lists some characteristics of effective agile test automation tools. These tools should:

Support starting the test automation effort immediately, using a test-first approach

Separate the essence of the test from the implementation details

Support and encourage good programming practices for the code portion of the test automation

Support writing test automation code using real languages, with real IDEs

Foster collaboration

Part III, “Using Agile Testing Quadrants,” and particularly Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” contain examples of test automation tools that work well on agile projects.

Implementing Automation

While you’re evaluating tools, think about how quickly your top priority automation need must be addressed. Where will you get the support to help implement it? What training does the team need, and how much time will be available to devote to it? How quickly do you have to ramp up on this tool?

Keep all of these constraints in mind when you’re looking at tools. You might have to settle for a less robust tool than you really want in order to get vital automation going in the short term. Remember that nothing’s permanent. You can build your automation effort step-by-step. Many teams experience unsuccessful attempts before finding the right combination of tools, skills, and infrastructure.

Selenium at Work

Joe Yakich, a software engineer with test automation experience, describes how a team he worked with implemented a test automation effort with Selenium, an open source test automation tool.

The software company I worked for—let’s call it XYZ Corp—had a problem. The product, an enterprise-level web-based application, was a powerful, mature offering. Development projects were managed using Agile and Scrum, and a talented stable of engineers churned out new features quickly. The company was growing steadily.

So, what was the problem? XYZ was facing a future where software testing efforts might not be able to keep pace with the development effort. Software quality issues might slow adoption of the product or—worse yet—cause existing customers to look elsewhere.

Test automation seemed like an obvious way to mitigate these risks, and XYZ was fully aware of it. In fact, they had attempted to create a test automation suite twice before, and failed.

The third time, XYZ chose to use Selenium RC, driven by the Ruby programming language. Selenium RC—the RC is for “Remote Control”—is a tool for test automation. Selenium RC consists of a server component and client libraries. The Java server component acts as an HTTP proxy, making the Selenium Core JavaScript appear to originate from the webserver of the application under test (AUT). The server can start and stop browser sessions (supported browsers include nearly all modern browsers, including Internet Explorer, Firefox, and Safari) and interpret commands to interact with elements such as buttons, links, and input fields. The client libraries allow test scripts to be written in Java, .NET, Perl, Python, and Ruby.

Our team chose Ruby because it’s a purely object-oriented, dynamic, interpreted language with a syntax that is elegant, expressive, and tersely powerful. Most importantly, Ruby is an ideal tool for the creation of a Domain Specific Language (DSL). Ruby is malleable enough for the programmer to first choose the structure and syntax of the DSL and then craft an implementation, as opposed to a more rigid language that might impose constraints on that freedom. One of our goals was to create an automation framework—a DSL—hiding complex detail. We wanted to be able to say things like

editor.save

in our tests instead of

s.click("//table[@class='edit']/tbody/tr[0]//img[@src='save.gif']")

Not only is the former more readable, it’s also far more maintainable. The XPath expression in the latter can be put in a library method to be called as needed. Using a DSL that employs the nouns and verbs of the application allows an engineer writing a test to focus on the test, not the underlying complexity of interacting with on-screen controls.

XYZ created an automation team to build the framework and tests. Creating the framework itself was a time-consuming, technically challenging task. Some of the framework classes themselves were complicated enough to warrant unit tests of their own. After a sufficient amount of test framework was constructed, we began work on actual application tests, using the Ruby RSpec library. RSpec is itself a DSL for test specifications. One of its strengths is the use of simple declarative statements to describe behavior and expectations. One might, for example, write a test using the statement

“A user should be able to save data in an editor by clicking Save”

filling in the body of the test with calls to the Selenium-based test framework we had created.

Nearly a year later, we had automated nearly two thousand test cases. Although the majority of the application was covered by automation, other portions of the application required manual testing—we had been forced to make choices and prioritize our efforts. Every week the test suite took longer to run than the preceding week; it now took nearly six hours to complete, and we had begun to think about running tests in parallel. We had not yet managed to expand our testing across all of the browsers supported by the application. The enthusiasm that automation generated had waned somewhat, and we found it necessary to carefully manage expectations, both with upper management and with other engineers. Despite these issues, Selenium was a clear win, for had we not invested heavily in test automation, testing at XYZ would have required hiring an army of test engineers (which would have been prohibitively expensive even had we been able to find enough qualified applicants).

Not everything can be automated, because of budgetary or technical reasons. In addition, exploratory testing is invaluable and should not be neglected. It should be noted, however, that these drawbacks are shared by every other test automation tool currently available, and most of the other automation tools that can rival Selenium’s automation prowess are commercial products that cannot match its price: free.

Good development practices are key to any automation effort. Use an object-oriented approach. As you build your library of test objects, adding new tests becomes easier. A domain specific language helps make business-facing tests understandable to customers, while lowering the costs of writing and maintaining automated test scripts.

Good object-oriented design isn’t the only key to building a suite of maintainable automated tests that pay off. You also need to run the tests often enough to get the feedback your team needs.. Whatever tools we choose must be integrated with our build process. Easy-to-interpret results should come to us automatically.

The tools we choose have to work on our platforms, and must share and play well with our other tools. We have to continually tweak them to help with our current issues. Is the build breaking every day? Maybe we need to hook our results up to an actual traffic light to build team awareness of its status. Did a business-facing test fail? It should be plain exactly what failed, and where. We don’t have extra time to spend isolating problems.

These concerns are an essential part of the picture, but still only part of the picture. We need tools that help us devise test environments that mimic production. We need ways to keep these test environments independent, unaffected by changes programmers might be making.

Building test infrastructure can be a big investment, but it’s one our agile team needs to make to get a jump on test automation. Hardware, software, and tools need to be identified and implemented. Depending on your company’s resources, this might be a long-term project. Brainstorm ways to cope in the short term, while you plan how to put together the infrastructure you really need to minimize risk, maximize velocity, and deliver the best possible product.

Managing Automated Tests

Let’s say we need a way to find the test that verifies a particular scenario, to understand what each test does, and to know what part of the application it verifies. Perhaps we need to satisfy an audit requirement for traceability from each requirement to its code and tests. Automated tests need to be maintained and controlled in the same way as production source code. When you tag your production code for release, the tests that verified that functionality need to be part of the tag.

Here’s an example where that comes in handy. We just found a problem in the code under development. Is it a new problem, or has it been lurking in the code for a while and somehow missed by the test? We can deploy the tag that’s in production, try to reproduce the problem, and investigate why the tests didn’t catch it. Lisa’s team recently had a situation where the regression suite missed a bug because a database constraint was missing in the test schema. That kind of problem is hard to pinpoint if you aren’t tying your test code versions to your production code versions.

Organizing Tests

Many tools come with their own means of organization. For example, FitNesse comes with its own wiki, with a hierarchical organization, and built-in version control. As of this writing, FitNesse is starting to provide support for source code control tools such as Subversion. Scripts written in other test tools, such as Watir and Canoo WebTest, can and should be maintained within the same source code control system as production code, just as the unit tests are.

Organizing Tests with the Project under Test

We asked some agile testing experts how they manage tests. Dierk König, founder and project manager of Canoo WebTest, explained how his teams have managed their automated tests to satisfy the needs of both the development and customer teams.

We always organize our tests alongside the project under test. That is, test sources are stored together with the project sources in the exact same repository, using the same mechanisms for revision control, tagging, and sharing the test base.

WebTest comes with a standard layout of how to organize tests and test data in directories. You can adapt this to any structure you fancy, but the “convention over configuration” shows its strength here. In large projects, every sub-project maintains its own test base in a “webtest” subdirectory that follows the convention.

Whenever a client did not follow this approach, the experience was very painful for all people involved. We have seen huge databases of test descriptions that did not even feature a proper revision control (i.e., where you could, for example, see diffs to old releases or who changed which test for what reason).

Keep in mind that tests are made up from modules so that you can eliminate duplication of test code; otherwise the maintenance will kill you. And before changing any module, you need to know where it is used.

In short: Make sure the master of your tests and your test data is in a text format that is versioned together with your code under test.

Nontechnical personnel (for example, management, QA) may require more high-level information about test coverage, latest test results, or even means of triggering a test run. Don’t let these valid requirements undermine the engineering approach to test automation. Instead, write little tools, for example, web-based report applications, that address these needs.

The ability of customers to access information about tests is as important as the ability to keep test and production code coordinated. As Dierk pointed out, you might not be able to do all this with the same tool.

Test management helps your team answer questions such as the following:

Which test cases have been automated?

Which still need automating?

Which tests are currently running as part of a regression suite?

Which tests cover what functional areas?

How is feature XYZ designed to work?

Who wrote this test case? When? Who changed it last?

How long has this test been part of the regression suite?

Because one of the primary reasons we write tests is to guide development, we need to organize tests so that everyone on the team can find the appropriate tests for each story and easily identify what functionality the tests cover. Because we use tests as documentation, it’s critical that anyone on either the development or customer team can find a particular test quickly when there’s a question about how the system should behave. We might need multiple tools to satisfy different test management goals.

It’s easy to lose control of test scripts. When a test fails, you need to pinpoint the problem quickly. You may need to know what changes have been made recently to the test script, which is easy with the history available in a source code control system. Your customer team also needs a way to keep track of project progress, to understand how much of the code is covered with tests, and possibly to run tests themselves. Test management systems, like the tests themselves, should promote communication and collaboration among team members and between different teams.

Test Transparency

Declan Whelan, a software developer and agile coach, uses a test management approach designed to keep tests visible to testers, developers, managers, and other teams.

We treat all test artifacts the same as source code from an organizational and revision control perspective. We use Subversion, and anyone who wants to run or edit the tests simply checks them out.

The latest Fit tests are available on a Confluence Wiki. We did this to support collaboration (team is distributed) and to leverage the strong capabilities of Confluence. Having the tests visible on the wiki was also helpful to others such as managers and other teams who did not want to check it out from the repository.

Prior to this, the QA team maintained test cases on a drive that was not accessible to anyone outside of QA. This meant that developers could not easily see what was being tested. Making the tests visible, transparent, and supported by a version control system (Subversion) really helped to break down barriers between developers and testers on the team.

Make sure your tests are managed with solid version control, but augment that with a way for everyone to use the tests in ways that drive the project forward and ensure the right value is delivered.

Organizing Test Results

Everyone involved with delivering software needs easy access to tests and test results. Another aspect of managing tests is keeping track of what tests are from prior iterations and need to keep passing, versus tests that are driving development in the current iteration and may not be passing yet. A continuous integration and build process runs tests for quick feedback on progress and to catch regression failures. Figure 14-4 shows an example of a test result report that’s understandable at a glance. One test failed, and the cause of the failure is clearly stated.

Figure 14-4 Test results from a home-grown test management tool

If you’re driving development with tests, and some of those tests aren’t passing yet, this shouldn’t fail a build. Some teams, such as Lisa’s, simply keep new tests out of the integration and build process until they pass for the first time. After that, they always need to pass. Other teams use rules in the build process itself to ignore failures from tests written to cover the code currently being developed.

As with any test automation tool, you can solve your test management problems with home-grown, open source, or commercial systems. The same criteria we described in the section on evaluating test tools can be applied to selecting a test management approach.

Test management is yet another area where agile values and principles, together with the whole-team approach, applies. Start simple. Experiment in small steps until you find the right combination of source code control, repositories, and build management that keeps tests and production code in synch. Evaluate your test management approach often, and make sure it accommodates all of the different users of tests. Identify what’s working and what’s missing, and plan tasks or even stories to try another tool or process to fill any gaps. Remember to keep test management lightweight and maintainable so that everyone will use it.

Managing Tests For Feedback

Megan Sumrell, an agile trainer and coach, describes how her team coordinates its build process and tests for optimum feedback.

We create a FitNesse test suite for each sprint. In that suite, we create a subwiki for each user story that holds its tests. As needed, we create a setup and teardown per test or suite. If for some reason we don’t complete a user story in the sprint, then we move the tests to the suite for sprint in which we do complete the story.

We scripted the following rule into our build: If any of the suites from the previous sprint fail, then the build breaks. However, if tests in the current sprint are failing, then do not fail the build.

Each test suite has a lengthy setup process, so when our FitNesse tests started taking longer than 10 minutes to run, our continuous integration build became too slow. We used symbolic links to create a suite of tests that serve as our smoke tests, running as part of our continuous integration build process. We run the complete set of FitNesse tests on a separate machine. We set it up to check the build server every five minutes. If a new build existed, then it would pull the build over and run the whole set of FitNesse tests. When it was done, it would then check the build server again every five minutes and after a new build existed, it would repeat the process.

Megan’s team took advantage of features built into their tools, such as symbolic links to organize FitNesse test suites for different purposes—one for a smoke test, others for complete regression testing. The team members get immediate feedback from the smoke tests, and they’ll know within an hour whether there’s a bug that the smoke tests missed.

Go Get Started

Don’t be afraid to get something—anything—in place, even if it’s somewhat deficient. The most important factor in success is to just get started. Many, if not most, successful teams have started with a poor process but managed to turn an inadequate process into something truly essential to the team’s success, one piece at a time. As with so many aspects of agile testing, improving in tiny increments is the key to success.

If you don’t start somewhere, you’ll never get traction on automation. Get the whole team together and start an experiment. Without the right level of test automation, your team can’t do its best work. You need the right test automation to deliver business value frequently. A year or two from now, you’ll wonder why you thought test automation was so hard.

Summary

In this chapter, we considered how to apply agile values, principles, and practices to develop an automation strategy. We discussed the following subjects related to automation:

Use the agile testing quadrants to help identify where you need test automation, and when you’ll need it.

The test automation pyramid can help your team make the right investments in test automation that will pay off the most.

Apply agile values, principles, and practices to help your team get traction on test automation.

Repetitive tasks, continuous integration and build processes, unit tests, functional tests, load tests, and data creation are all good candidates for automation.

Quadrant 3 tests such as usability testing and exploratory testing may benefit from some automation to set up test scenarios and analyze results, but human instincts, critical thinking, and observation can’t be automated.

A simple, whole-team approach, using iterative feedback, and taking enough time can help you get started on a good solution.

When developing an automation strategy, start with the greatest area of pain, consider a multi-layered approach, and strive for continuously revisiting and improving your strategy rather than achieving perfection from the start.

Consider risk and ROI when deciding what to automate.

Take time to learn by doing; apply agile coding practices to tests.

Decide whether you can simply build inputs in-memory, or whether you need production-style data in a database.

Supply test data that will allow tests to be independent, rerunnable, and as fast as possible.

Take on one tool need at a time, identify your requirements, and decide what type of tool to choose or build that fits your needs.

Use good development practices for test automation, and take time for good test design.

Automated tools need to fit into the team’s development infrastructure.

Version-control automated tests along with the production code that they verify.

Good test management ensures that tests can provide effective documentation of the system and of development progress.

Get started on test automation today.