Agile Testing: A Practical Guide for Testers and Agile Teams

Chapter 9 Toolkit for Business-Facing Tests that Support the Team

In the previous chapter, we talked about how to approach business or functional testing to support the team in its effort to build the right software. In this chapter, we’ll examine some of the tools you can use to help your team succeed with Quadrant 2 tests.

Business-Facing Test Tool Strategy

How do we capture the business-facing tests that help the programmers know what to code? Face-to-face conversations between programmers and customers are usually the best way, but even when customers are part of your team, they don’t have all day to hang out with programmers and explain features. If any customer or developer team members are in different locations, impromptu hallway conversations might not be feasible. Besides, six months from now, we might want a way to remember why we coded a piece of functionality a certain way. If some of our team members are in different locations, we’re definitely going to need some way to share information electronically.

As agile development has gained in popularity, we have more and more tools to help us capture examples and use them to write executable tests. The tools available are changing too fast for us to include an inventory of them in this book, but we can offer some examples of tools and some strategies for using them to help provide business-facing tests that support the team’s development of new stories. Some of the tools we discuss here aren’t new, or specific to agile development, but they work well in an agile project.

Your strategy for selecting the tools you need should be based on your team’s skill set, the technology your application uses, your team’s automation priorities, time and budget constraints, and other concerns unique to your situation. Your selection of a tool or tools should not be based on the latest and coolest tool offered by a salesman. You might need many different tools to solve different problems.

For more information about a general approach to test automation, see Chapter 14, “An Agile Test Automation Strategy.”

We encourage customers to do some advance preparation and to be ready to explain examples for each story during iteration planning. Testers are in a good position to help customers figure out how to provide the right amount of detail at the beginning of the iteration. It’s hard to strike just the right balance.

Lisa’s Story

Soon after our team chose to use FitNesse for specifying and automating business-facing tests, our product owner and I tried to make good use of the new tool. We had an extremely complex epic coming up. We spent many hours writing detailed test cases for highly complex business rules weeks in advance of the iteration where the first story of the epic was started. We felt good about getting a running start on developing the new functionality.

When they started working on these stories, the programmers complained that they couldn’t get the big picture from these detailed tests. The tests were also designed in a way that was incompatible with the actual code design. I ended up spending hours refactoring them. It wasn’t a complete waste of time, because at least I understood the stories well and we had a number of test cases we could use eventually, but it wasn’t the right approach for our team. Trial and error has shown us that high-level tests combined with a few examples of desired and undesired behavior are the best way for the programmers to know what to start coding.

—Lisa

Experiment with different levels of up-front detail in test cases to figure out what works best for your team. Whatever level of detail you’re after, you need some way to help customers find and express examples of desired system behavior. In the next section, we look at the types of tools that can do that.

Tools to Elicit Examples and Requirements

As we pointed out in Chapter 8, stories are only a starting place for a prolonged conversation about the desired behavior. Having correctly sized stories where the feature, user, and purpose are clearly stated gives us a head start. They aren’t very detailed, because as Mike Cohn [2004] points out, it’s best to defer collecting details until the story is included in an iteration. Collecting details for a story that might never be included is a waste of resources. We like the “role, function, business value” pattern for user stories that Mike Cohn describes in User Stories Applied, as in:

As a (role), I want (function) so that (business value).

This format doesn’t work for everyone, so we encourage you to experiment and see what works best in your situation. Regardless of how your user stories read, you need some way to flesh those stories out with examples and business-facing tests that guide development.

One simple story can have a wide-ranging impact, not only on the application, but across the organization, its clients, its associates, vendors, or partners. If we change an API, we have to notify any customers or vendors who might be using it. If we plan a UI change, we want, or might even be contractually obligated, to give a certain amount of advance notice to users. Stories may affect legal concerns or impact external reporting. New features often mean new or updated documentation. Of course, changed functionality is likely to affect other parts of the system.

The software development team, including the testers, should help the customer capture and communicate all of the requirements related to each story or theme. Developing new features, only to be prevented from releasing them for legal reasons or because a business partner wasn’t informed in time, is a frustrating waste of time (just ask Lisa!). Lean development teaches us to avoid waste while we develop software.

What tools can help us illustrate desired behavior with examples, brainstorm potential implementations and ripple effects, and create requirements we can turn into tests? Some examples are:

Checklists

Mind Maps

Spreadsheets

Mock-Ups

Flow Diagrams

Software-Based Tools

The list includes a number of simple tools that aren’t unique to agile testing but that shouldn’t be neglected. In agile development, simple solutions are usually best. Let’s look at these in more detail.

Checklists

Checklists are one way for product owners to make sure they correctly assess and communicate all of the aspects of a story. The product owner for Lisa’s team, Steve Perkins, came up with his own “story checklist” to make sure he and the stakeholders think through everything affected by the story. He created a template on the team wiki for this purpose. The checklist specifies the conditions of satisfaction—what the business needs from the story. It also includes impacts on existing functions such as the website, documents, administrative forms, account statements, and other components of the system and the daily operation of the business. The checklist makes sure the team doesn’t miss requirements such as data migration, notifications, legal considerations, and communications to vendors and business partners because they forgot to consider them. Figure 9-1 shows a sample story checklist.

Figure 9-1 Sample story checklist

Mind Maps

Mind maps are a simple but effective way to search out ideas that that might not occur to you in a simple brainstorming session. Mind maps are diagrams created to represent concepts, words, or ideas linked to a central key concept. We used mind maps to organize this book.

It really doesn’t matter whether you purchase a tool such as the one we used or draw on a whiteboard or a big piece of paper. The effect is the same. Mind maps enable you to generate ideas and work in a way that is consistent with the way you think about problems.

How about an example? We’re discussing the story shown in Figure 9-2.

Figure 9-2 Shopping cart delete story

We gather around the whiteboard and start asking questions. Where should the deleted items go? Should they be saved for later approval, or should they just disappear? What should the screen look like after we delete an item? Figure 9-3 shows an example of the sort of mind map we might draw on a whiteboard.

Figure 9-3 Example mind map for shopping cart delete story

Spreadsheets

When possible, tools for specifying business-facing tests should fit well with your business domain. For example, spreadsheets are widely used by financial services companies, so for a project in the financial services area it makes sense to use spreadsheets to define examples of the functionality that a story should deliver.

Customers can write a few high-level test cases to help round out a story prior to the start of the iteration, possibly using some type of checklist. Some customer teams simply write a couple of tests, maybe a happy path and a negative test, on the back of each story card. Some write more detailed examples in spreadsheets or whatever format they’re comfortable working with.

Steve Perkins, the product owner for Lisa’s team, often illustrates complex calculations and algorithms in spreadsheets, which the team can turn into tests later. Figure 9-4 shows one of his worksheets, which performs calculations on the input values to produce the values in the ADR and ACR columns. This format is easy to get into an automated test framework (refer to Figure 9-8 for the corresponding FitNesse example).

Figure 9-4 Spreadsheet example from product owner

Look at tools already used by your business experts and see whether they can be adapted to document examples of desired feature behavior to help the development team better understand the story.

Janet has worked with several teams that have used spreadsheets as input into their Fit tests. This allows customers to work in a tool that is familiar to them but not waste any effort in translating them to an automation tool.

Mock-Ups

Mock-ups can take many forms. Paper prototypes are a simple but effective way to test how screens will work together. Drawing on a whiteboard can accomplish the same goal, but it can’t be passed around. Screenshots from existing applications can form the basis of a discussion about how to add a new feature and where it will fit into the UI. You may have used tools like these in other development methodologies. The big difference in agile development is that we create and discuss the mock-ups just as we’re about to start writing the code, rather than weeks or months beforehand. We can be confident that the mock-up represents what the customers want right now.

See Chapter 8 for Gerard Meszaros’ description of using paper prototypes and Wizard of Oz testing.

Lisa’s Story

We use simple approaches to creating mock-ups so that we aren’t tempted to invest time coding before we’re finished working through the mock-up. Often, we draw a UI or workflow on the whiteboard and then take photos of it to upload to our team wiki so our remote team member can also see it. At other times, a customer or our product owner draws the mock-up on paper or modifies an existing UI page or report to show what should be added and changed. The paper mock-ups are scanned in and posted on the wiki.

A picture’s worth a thousand words, even in agile software development. Mock-ups show the customer’s desires more clearly than a narrative possibly could. They provide a good focal point for discussing desired code behavior.

—Lisa

Figure 9-5 shows an example of a mock-up that Lisa’s team used to mock up a new report—simply by marking up an existing report that’s similar.

Figure 9-5 Sample report mock-up

Mock-ups don’t need to be fancy or pretty, or to take a lot of time to create. They do need to be understandable to both the customer and developer teams.

Flow Diagrams

Simple diagramming tools are helpful, whether the team is co-located or not. It’s often a good idea to capture in a more permanent form a workflow or decision tree worked out during a discussion. Flow diagrams can become the basis of a user scenario that might help you tie two or three user stories together. Let’s look at the shipping order story again that we introduced in Chapter 8 (see Figure 9-6).

Figure 9-6 Story for shipping charges

Figure 9-7 shows a very simple flowchart of a decision process for whether a customer’s order is eligible for free shipping based on a threshold order amount. Because we’ve discussed this story with our customer, we’ve found out that the customer’s order must not only exceed a threshold dollar amount but also must be to one address only, and it must weigh less than a shipping weight threshold. If all of these conditions are satisfied, the customer’s order will ship free; otherwise, the customer will have to select from the “choose shipping options” page.

Figure 9-7 Flow chart for qualifying for free shipping option

Visuals such as flow diagrams and mind maps are good ways to describe an overview of a story’s functionality, especially if they’re drawn by a group of customers, programmers, and testers. In agile development, we create these diagrams as we’re about to start writing tests and code. From these, the team can immediately start digging down to the detailed requirements.

Software-Based Tools

If we’re in a different location than our customers, we need tools to help us converse with them. Distributed teams tell us that desktop sharing is the number one tool that helps them deal with working in separate locations. Windows NetMeeting and VNC are examples of tools that let two team members in different locations pair-test. Video conferencing tools such as WebEx and Skype enable collaboration and demos between remote teams and customers. Online whiteboards such as Scriblink and interactive whiteboard tools such as Mimeo facilitate distributed whiteboard discussions.

More tools that are geared for direct use by product owners and business experts are becoming available, and many teams develop their own. Tools such as Fit (Framework for Integrated Tests) and FitNesse were designed to facilitate collaboration and communication between the customer and development teams. We’re hearing about more teams where the customers actually write the tests in a tool such as those.

Notes from a Distributed Team

Pierre Veragen and Erika Boyer of iLevel by Weyerhaeuser told us that every iteration begins with everyone on the team writing acceptance tests. That’s how they start their iteration planning. Most interesting is the fact that their product owners, who are mechanical engineers, write FitNesse tests themselves. Pierre explains that an advantage of a tool such as FitNesse is the ability to use their own domain language in the FitNesse tests. It doesn’t matter what they end up choosing as a UI. They can test all of their complex calculations in the tests.

With this process, tests can be written before writing the testing code or the system under test. It’s true test-driven development. Behavior changes and bug fixes can follow.

Some teams build their own frameworks that allow customers, business analysts, and testers to document examples that can be directly turned into executable tests. These are often based on open source tools such as xUnit, Fit, Selenium, and Watir. We like this approach, because it saves time and resources. When you’re delivering production-ready code in short iterations, you need a streamlined process.

Online forum tools are a good alternative to email conversations for ongoing discussions about features or technical concerns, especially for teams that don’t all sit together. Emails often get missed or lost, people have to remember to choose “Reply all,” and it can be hard to put together the details of the discussion later. Lisa’s team uses an online forum to elicit opinions about different tools, propose different behavior for features, and conduct philosophical discussions such as whether to track defects.

Finding the right electronic tools is particularly vital for distributed teams. Instant messaging, the telephone, VoIP, and Skype help us communicate, but they lack the visual component. Some global teams ask their members to meet at nonstandard hours so that they can have real-time conversations, but frameworks for written and visual communication are still critical.

Wikis are a common tool used to enhance communication and record discussions and decisions. Wikis enable users to edit web page content in a web browser. Users can add hyperlinks and easily create new pages. You can upload mock-ups, samples, and pictures of whiteboard drawings and make them easily visible on Wiki pages. The hierarchical organization can get tricky to maintain, but there are lots of open source and vendor wiki software packages available that make managing your knowledgebase and sharing information easier to administer. If your wiki knowledgebase has grown to the point where it’s hard to find anything, hire a technical writer to transform it into organized, usable documentation.

Open source and commercial tools provide ways to let teams collaborate on requirements and test cases online. We can’t emphasize enough the need for you to identify tools that might be helpful, to experiment with them for a few iterations, and to decide how well they work for you. Your team’s needs will change with time, so always be open to trying new techniques and frameworks.

These tools help create the conversation about the story. With these techniques, and as much real-time conversation and visual sharing as we can manage, we can define the right product from the get-go.

Tools for Automating Tests Based on Examples

What about test tools? We like the collaboration inherent with tools such as Fit and FitNesse. However, in our opinion, any tool that gets testers and programmers, programmers and customers, and testers and customers talking is a great one. We know teams where customers actually write tests in Fit, FitNesse, Expect, or other tools. This works when the tool has been set up in a manner that’s clear to everyone writing tests, with the domain language easy to understand and the appropriate fixtures provided.

Tools to Test below the GUI and API Level

There are a multitude of open source tools that enable you to test below the GUI or at the API layer. We are listing just a few, but your team will need to determine the right tool for you.

Unit-Level Test Tools

Some teams use the xUnit tools such as JUnit or NUnit for business-facing tests as well as technology-facing tests. If the testers and customers are comfortable with these tools, and they provide for all of the functional testing behind the GUI needed, they’re fine. To make these tools more customer-friendly, teams might build a framework on top of the unit-level tools that testers and customers can use to specify tests.

Janet has worked on a couple of applications like that. One was a message handling system that was being deployed in an organization. The programmers used JUnit for all of the component and integration testing. They built a load test framework that could make use of the JUnit tests, so no other testing tools were needed. The GUI front end was so small that Janet was able to test it manually. It made no sense to automate the GUI testing in this case.

Behavior-driven development (BDD) tools are also suited to this purpose, because they use a more natural language for specifying the tests. Behavior-driven development is a variation of test-driven development, pioneered by Dan North [2006], and evolved by many others. It’s related to domain-driven design, with a focus on the domain rather than on the technology, and driving design with a model. Instead of the word “test” or “assert,” BDD uses the word “should.” By thinking in terms of behavior, it’s natural to write specifications ahead of code. Test specifications use a domain-specific language to provide tests that customers can read but that can also be easily automated.

Some of the many BDD tools available as of this writing include easyb and JBehave for the Java platform, NBehave and NSpec for .NET, and RSpec for Ruby. These tools, like the XUnit tools, are intended for use by programmers to guide coding, but they can also be used to express business-facing tests that drive development, involving customers more closely in the development process.

Behavior-Driven Development

Andrew Glover, president of Stelligent Incorporated and author of books including Continuous Integration and Java Testing Patterns, explains the thinking behind one of the BDD tools, easyb.

assertEquals(42.50, order.price(), 0.0). Without examining the context in which this statement appears, this code is somewhat incomprehensible. Now imagine you don’t even read code—that is, you are a stakeholder asking (actually paying) for new features. The previous code statement might as well be Farsi (assuming you can’t actually read Farsi!).

order.price().shouldBe 42.50. While the context in which this statement appears is still absent, this line of code is a bit more coherent. In fact, it reads like a normal sentence (and this time knowledge of Farsi isn’t required!). Stakeholders, in this case, could understand this code if they chose to read it; on top of that, it turns out that this line of code essentially matches what they asked for in the first place. This line of code describes behavior in a more literal manner too—the code uses a normal everyday phrase like shouldBe, which is distinctly different than the previously written assertEquals.

Both lines of code from the previous paragraphs convey the same meaning and indeed validate the same requirement, yet the latter one comes awfully close to leveraging the customer’s language. This is a fundamental point of the notion of behavior-driven development, which strives to more appropriately validate a software system by thinking in terms of the term “should” rather than test. In fact, by focusing on behavior and closely modeling behavior after what stakeholders ask for, behavior-driven development converges on the idea of executable documentation. Indeed, through leveraging a stakeholder’s language, there is a decreased impedance mismatch between what he wants and what he ultimately receives; moreover, employing a stakeholder’s language facilitates a deeper level of collaboration between all parties. Listen to how a conversation might go:

Stakeholder: For the next release of our online store, our Gold-level customers should receive a discount when they make a purchase.

Developer: What kind of discount—what criteria do they have to meet in order to receive it?

Stakeholder: When they have at least $50 dollars in their shopping cart.

Developer: Does the discount increase based upon the amount, or is it fixed regardless of the value of the shopping cart?

Stakeholder: Good question—the discount is fixed at 15% regardless of price. So, given a Gold-level customer, when the shopping cart totals $50 or more, it should receive a 15% discount off the total price.

The last statement of the stakeholder is key—note how the requirement has been specified and the means for validating it. In fact, the stakeholder has essentially narrated a specific scenario in a larger story related to discounts.

Given this scenario, a developer can take the stakeholder’s comments—word for word—and execute them. For example, one behavior-driven development framework, dubbed easyb, facilitates system validation through a domain-specific language that supports stories and scenarios. For example:

scenario "Gold-level customer with $50 in shopping cart", {

given " a Gold-level customer"

when "their shopping cart totals $50 or more"

then " they should receive a 15% discount off the total price"

}

Of course, this particular scenario doesn’t actually do anything (other than capturing the stakeholder’s requirements, which is still quite important!); consequently, it is considered pending. This status alone conveys valuable information—stakeholders can, first and foremost, see their words as a means to validate their requests, and secondly, gauge if their requirement has been fulfilled. After this scenario has been implemented, it can, of course, take on two other states—success or failure, both of which serve to convey further status information to interested parties.

Now, with a collaborative scenario defined, development can proceed to the implementation—the beauty in this case is that they can directly implement the desired behavior inline with the requirements, like this:

scenario "Gold-level customer with $50 in shopping cart", {

given "a Gold-level customer", {

customer = new GoldCustomer()

}

when "their shopping cart totals $50 or more", {

customer.shoppingCart << new Item("widget", 50.00)

}

then "they should receive a 15% discount off the total price" , {

customer.orderPrice.shouldBe 42.50

}

}

This scenario is now executable within the context of the application it serves to validate! The scenario leverages the customer’s exact words, too; what’s more, regardless of the customer’s ability to read code, the code itself leverages natural language: customer.orderPrice.shouldBe 42.50.

By leveraging the customer’s language, the customer has the ability to collaboratively facilitate in validating the system he or she wants built. Also, with development leveraging the stakeholders’ language, there is a direct link between what stakeholders ask for and what they receive. And you don’t even need to understand Farsi to see the benefit in that.

Two of the most common questions we’re asked by new agile teams are, “What about documentation?” and “How can test automation keep up with development in two-week iterations?” Tools such as easyb answer that question with executable documentation using a domain-specific language that everyone on both the customer and developer teams understands.

The goal of business-facing tests that support the team is to promote communication and collaboration between customers and developers, and to enable teams to deliver real value in each iteration. Some teams do this best with unit-level tools, and others adapt better to functional-level test tools.

API-Layer Functional Test Tools

Before Lisa joined her first agile team, testing “behind the GUI” was a concept that sounded good, but she’d never had the opportunity to try it. Fit, and FitNesse, which is built on top of Fit, are functional test tools that grew from the need for the customer team to be able to write and understand the business-facing tests that drive development. With these tools, teams can test business logic without involving the presentation layer.

Fit and FitNesse. Fit (Framework for Integrated Tests) is an open source testing framework that promotes collaboration, which makes it a good tool to help refine requirements. The invention of Ward Cunningham, Fit has enjoyed an illustrious roster of contributing developers. Fit enables customers, testers, and programmers to use examples to specify what they expect the system to do. When the tests run, Fit automatically compares customers’ expectations to actual results.

With Fit, customers can provide guidance using their subject matter expertise to define the examples that the programmers can code against. The programmers participate by writing the fixtures that do the actual checks against the examples. These fixtures use the data specified in the examples to run with the actual program.

Fit tests are automated by fixtures that pass the test inputs to the production code and then accept the outputs, which it then compares with expected results. The test results are color-coded, so it’s easy to spot a failure or exception.

Fit tests are written as HTML tables, but teams can customize Fit so that tests can be written in spreadsheets or whatever form the customers, testers, and analysts find usable.

Learn more about Fit at fit.c2.com.

FitNesse is a web server, a wiki, and a software testing tool that is based on Fit. Originally developed by Robert C. “Uncle Bob” Martin and Micah Martin, it’s an open source tool with an active developer community. The main difference between FitNesse and Fit is that FitNesse tests are written in wiki markup instead of HTML tables, which some users find easier. It also supports creating tests in spreadsheets and importing those into the tests.

Learn more about FitNesse at www.fitnesse.org.

Figure 9-8 shows part of the FitNesse test that was built from the example in Figure 9-4. More inputs were added to make the production code run, but the essential test data is from the spreadsheet. The test results are color-coded green when they pass, red when they fail.

Figure 9-8 Automated FitNesse test from customer example

Another benefit of a Fit or FitNesse type of tool is that it promotes collaboration among different team members in order to come up with the right tests to guide development. Customers, programmers, testers, and others work together to specify and automate the tests.

Testing Web Services. Web services is just another form of an API that enables other applications to access your application. Let’s talk about some of the tools you can use to test various inputs into your system.

CrossCheck. CrossCheck is one example of a tool for testing web services. You supply the WSDL (Web Services Description Language); CrossCheck compiles the page and then presents you with a tabbed menu that contains textboxes for you to fill in. It has a Run mode where you can add your tests to a suite and then run the suite. Neither Lisa or Janet have tried this tool, but it was noted on the Yahoo agile-testing group as a tool to use for testing web services if you were running the same data through each time.

Ruby Test::Unit. One project Janet was on used Ruby’s unit testing framework, Test::Unit, to test web services, with great success. In fact, the team was able to test early to give the programmers immediate feedback, which helped with the final design.

See the “System Test” example in Chapter 12, “Summary of Testing Quadrants,” to see how Janet’s team used Ruby Test::Unit to test web services.

soapUI.Another tool suggested for testing web services is soapUI. It has a steep learning curve but can be used for performance and load testing. Because it can loop though rows in an Excel spreadsheet or text file, it can be used for data-driven testing.

Tests that work at the layers below the presentation layer are well suited for writing and automating customer tests that guide coding. Some practitioners haven’t gotten the value they expected from story test-driven development. Brian Marick [2008] hypothesized that an application built with programmer test-driven development, example-heavy business-facing design that relies heavily on whiteboard discussions, a small set of automated sanity tests, and lots of exploratory testing could be a less expensive and equally effective approach. Whichever approach you take, if you’re testing an application with a user interface, you’ll need some automation at the GUI level.

Tools for Testing through the GUI

Wait a minute. How can we use GUI tests to drive development, because the GUI won’t be ready until the story is complete? It sound counterintuitive, but automated GUI tests are important to help us while we’re developing new functionality. Test frameworks can be used to specify test cases for a GUI tool before the code is written. In addition, you can automate GUI tests before coding is finished, either by using HTML mock-ups or by developing an end-to-end bare-bones slice through all of the screens that simply navigates but doesn’t provide all of the functionality yet. Even if you’re not using a lot of automated story tests to drive development, manual exploratory testing that helps us learn about the functionality and provides immediate feedback gets pretty tedious and slow without any assistance from automation. Let’s look at the types of GUI test tools that help drive development using business-facing tests.

A Tool Selection Rationale

David Reed, a test automation engineer, and his team went with soapUI Pro to automate testing for their web services. Here are some reasons he gave for choosing this particular tool.

• It has an open source version, so you can try it out it for free. You can learn it, kick the tires, expand stuff, and learn its strengths and weaknesses.

• It was easy to figure out what requests to make for what service.

• The assertions provided for verifying the results from requests are great and expandable. One really helpful one is verifying that the response comes back in an acceptable amount of time, raising an error if it doesn’t.

• The Pro version takes a lot of the hassle out of designing XPath queries to verify results. It also adds some nice touches for retrieving database data.

• It’s expandable with Groovy, a Java-based scripting language. (They’re working on a Java application, so it pays to have Java-friendly tools.)

• Developers can use it without sneering at it as a “test tool.”

• It’s easily integrated with our continuous integration environment.

• It has a feature to check code coverage.

• The price is right.

Record/Playback Tools

Record/playback tools are appealing because you can usually learn how to record a script and play it back quickly, and you can create lots of scripts in a short time. However, they have drawbacks. Early GUI test tools recorded mouse movements using X-Y screen coordinates. Scripts using those tools might also be sensitive to changes in screen resolution, color depth, and even where the window is placed on the screen.

Most modern GUI test tools use objects to recognize the controls in a graphical application, like buttons, menus, and text input widgets, so they can refer to them symbolically rather than with raw screen coordinates. This makes the application much more testable, because it’s more robust standing up to changes. A button might move to a different part of the screen, but the test can still find it based on its object name.

Even with improved object recognition, scripts created with record/playback are usually brittle and expensive to maintain. Recording can be a good way to start creating a script. Testers or programmers who know the tool’s scripting language can refactor the recorded script into an object-oriented model that’s easier to use and maintain. Historically, record/playback tools used proprietary scripting languages, which programmers aren’t interested in learning. It’s also more difficult to change the design patterns used in the tests.

Some script-based tools such as the ones we’ll talk about in the next few sections offer a record feature to help people get a quick start on writing the test script. However, with those tools, the recorded scripts aren’t intended for straight playback; they’re just a starting point to creating a well-designed and easily maintained suite of tests.

Many agile teams prefer tools and scripting languages that let them create their own domain-specific language (DSL). This makes tests much easier for business experts to understand and even write. Let’s look at some of these next.

Agile Open Source Test Tools

Each of the tools in this section was originally written by an agile development team that needed a GUI test tool and couldn’t find any third-party tools that worked for its situation. With these tools, you can write scripts that use web applications just like a human user. They fill in text fields, select from lists, and click checkboxes and buttons. They provide a variety of ways to verify correct navigation and contents of pages, such as tool-specific verify steps or XPath. Some of these tools have a higher learning curve than simple record/playback tools, but the extra investment of time usually pays off in scripts with a low total cost of ownership.

Ruby with Watir. Watir (Web Application Testing in Ruby) is a simple open source Ruby library for automating web browsers that works with Internet Explorer on Windows. There are different flavors for other browsers, including FireWatir for Firefox and SafariWatir for Safari.

Janet’s Story

I worked on a project that developed a three-layer test framework using Ruby and Watir. The first layer was a common set of libraries, and the second layer was to access the pages and provide navigation. The third and top layer created a domain language using fixture-type methods that mapped to the business needs. This allowed the manual testers to write high-level automated tests for workflows before coding was completed. If a fixture didn’t exist because of new functionality, the test could be created and the action word for the missing fixture could be “dummied” in. As soon as the fixture was coded, the test could be run as an acceptance test.

A very simple example of using Ruby with Watir incorporates the idea of DSL. Methods were created to simplify the tests so that any of the testers could actually create an automated script without knowing any Ruby or Watir.

This next example shows a test, and then two of the methods used in the test.

def test_create_new_user

login 'administrator','admin'

navigate_to_tab 'Manage Users'

click_button "Create New User"

set_text_field "userFirstNameInput", "Ruby"

set_text_field "userLastNameInput", "RubyTester"

click_button "Save Changes"

verify_text "Saved changes"

end

# methods created to support easier test writing

def navigate_to_tab(menuItemName)

@browser.link(:text,menuItemName).click

end

def set_text_field(id, value)

@browser.text_field(:id,id).set value

end

A third level could easily be added if create_new_user was called more than once. Just extract the common code that the test could call:

create_new_user (Ruby, RubyTester)

These tests were well suited to guiding development and providing quick feedback. Making tests easy for testers and customers to write, while keeping the automation framework designed for optimum maintainability, reduced the total cost of ownership of the tests.

—Janet

There are always drawbacks to any tool you use. For example, there are limitations to using objects. Sometimes programmers use custom controls or a new toolkit that your tool might not understand.

Janet’s Story

I started a new job as QA manager, and after much deliberation we decided to drop the vendor tool that the team had been using for a couple of years. We could not figure out what tests were actually being run, or what the real coverage was. We decided to start automating the tests using Ruby and Watir. The automation went fairly quickly at first, but then the tests started failing. We spent a lot of time changing the tests to reflect new object names. The developers were just using the default WebLogic object names, which would change every time a new object was added to the page. The testers went to the developers to ask if they could change the way they were coding. It took a little convincing, but when the developers realized the problems their practice was causing, they changed their habits. Over time, all of the defaults were changed, and each object had an assigned name. The tests became much more robust, and we spent much less time in maintenance mode.

—Janet

Implementing a new test automation tool usually requires some experimentation to get a good balance of testable code and well-designed test scripts. Involving the whole team makes this much easier. Watir is one example of a GUI test tool that we’ve found is well suited to agile projects. Let’s look at a couple more, Selenium and Canoo WebTest.

Selenium. Selenium is another open source tool, actually a suite of tools, for testing web applications. The tests can be written as HTML tables or coded in a number of popular programming languages, and can be run directly in most modern web browsers. A Firefox plug-in called “Selenium IDE” provides a way to learn the tool quickly. A recorder is provided to help create the tests, including writing assertions. Tests can be written in several different common programming and scripting languages, including Java, C#, and Ruby.

See Chapter 14, “An Agile Test Automation Strategy,” for an example of using Selenium RC to create a domain-specific test automation framework.

Canoo WebTest. In WebTest scripts, tests are specified as “steps” in XML files, simulating a user’s actions through a web UI. Here’s an example of how a WebTest script might invoke a page and verify the results:

Rather than driving an actual browser, as Selenium and Watir do, WebTest simulates the desired browser using HtmlUnit. The advantage of specifying tests as opposed to coding test scripts, is because there’s no logic in them, you don’t have to test the test.

Lisa’s Story

My team chose WebTest to automate smoke tests for our legacy application for several reasons. Because the scripts are written in XML, the programmers on the team were comfortable using the tool. It uses Ant to run the tests, so integrating it into the continuous build process was simple. It’s easy to learn, and the tests can be designed in a modular fashion, so they’re fairly easy to maintain. WebTest supports testing PDF files, emails, and Excel files, all of which are widely used in our application.

Being accustomed to powerful commercial test tools, I was skeptical of the concept of specifying tests, as opposed to programming them. I was amazed at how effective the simple tests were at catching regression bugs. It’s possible to put logic into the tests using Groovy or other scripting languages, but we’ve only found the need in a few cases.

Writing a few tests per iteration, I automated smoke tests for all of the critical areas of our application in eight months. These simple tests find regression bugs regularly. We refactor the tests frequently, so they are relatively easy to maintain. Our ROI on these tests has been tremendous.

—Lisa

Selenium, WebTest, and Watir are just three examples of the many open source tools available for GUI testing as of the time we wrote this book. Many teams write their own test automation frameworks. Let’s look at an example in the next section.

“Home-Brewed” Test Automation Tools

Bret Pettichord [2004] coined the term “home-brewed” for the tools agile teams create to meet their own unique testing needs. This allows even more customization than an open source tool. The goal of these tools is usually to provide a way for nontechnical customer team members and testers to write tests that are actually executable by the automated tool. Home-brewed tools are tailored to the exact needs of the project. They can be designed to minimize the total cost of ownership. They’re often built on top of existing open source tools.

Janet has been involved in a few projects that have used Ruby and Watir to create a full framework for functional testing. These frameworks allowed customers to specify tests that were then turned into a functional regression suite.

No test tool guarantees success. In fact, the history of test automation is littered with failed attempts. Having the whole team think about the best tools to use is a big help, but no matter what tool you use, you need a smart approach to writing tests. We’ll discuss that in the next section.

PAS Functional Testing

This next story is about one project Janet worked on that enjoyed success with home-brewed test automation.

PAS is a production accounting application for the oil and gas industry. Using gross meter readings and contract agreements, it must calculate ownership of the various products down to a very precise level (i.e., the components in gas). There are literally thousands of interactions between the combinations available in configuring the system and the actual outputs visible to a user. Given the large number of interactions, PAS has employed many complementary strategies for testing.

Joseph King, one of the initial programmers and agile coach for the team, tells us the story of how they accomplished their functional testing.

At the lowest level, there are developer functional tests that exercise specific functions via an API and verify the results using another read-only user API. There are currently over 24,000 tests implemented in JUnit that every developer must run before they can “check in” their changes to the source code.

The next level is a set of GUI tests that test the marshalling of user data back and forth to the API, particularly around “master-data” creation and updates. There are currently over 500 of these tests implemented using Watij (an open source library similar to Watir but using Java) and JUnit that run multiple times a day.

The final level of testing is a set of integration tests created by the users that run in a Fit-like harness. Users identify dense test cases that reflect real-world cases covering many of the functions that work together to produce financial and regulatory outputs. These test cases are then transcribed into import templates and then processed using a domain language that mirrors the way end customers think about their processes.

For example, after an end customer has created the configuration of facilities and contracts they wish to exercise in their test, they work with a developer to use the domain language to process their facilities in the correct order. The end users also supply a set of expected outputs that are then verified using a read-only API. These outputs can contain thousands of numbers, any of which can change for seemingly minor reasons in an evolving product. It is a constant challenge to sort through what is a legitimate business change from what is a defect. There are currently over 400 integration tests, and they run twice per day, providing feedback to the end customers and developers.

Exploratory testing is done continuously throughout the development cycle and is augmented at the end of releases.

Our first attempt at PASFIT (which is what we called the functional test framework) was a spreadsheet of color-coded inputs and outputs. We then generated Java code based on the color of the cells to create the data in PAS. That proved difficult to maintain, partly because the application was in major flux both at the GUI and database level.

Our next iteration of PASFIT didn’t evolve for nearly a year after the previous attempt. After we had a more stable set of database views and GUI, we were able to create an engine that used simple imperative language (i.e., a script) to do actions with arguments against a GUI (e.g., Go to Balancing Page, Balance Battery: Oil, Water). The script evolved into following the thought process of a production accountant and became a domain-specific language. The engine was written using Ruby and Watir, and an instruction from the script was basically a Ruby method that was invoked dynamically so that it was easy to update. After the script ran, the framework then loaded a snapshot of the views that the test wished to compare and did a simple row-by-row, cell-by-cell comparison of what was to be asserted and what actually happened. Eventually this was enhanced in the spreadsheet to use Pivot tables to enable the users to focus in on only the results they wished to assert for their test. All in all it has been quite successful, although the requirements for our application mean that 300 tests take about 12 hours to run, which is a long time.

Getting the business more involved in maintaining the regression tests has also been difficult, but when it happens it is very good. Currently, we have a stand-up where the business users and the developers meet for 15 minutes to pick up any of the scenario tests that are breaking that day. It is quite effective in that people often know when they come to the stand-up what they might have broken the day before. Future enhancements are likely to include asserting against actual user reports instead of the views and running a migration each night against the scenario script.

PASFIT achieved a balance between letting business experts write tests in a DSL and automating those tests with a highly complex application. Success came with some trial and error. Teams that write their own test frameworks need time to experiment to find the right solution for both the business and the development team.

Strategies for Writing Tests

The best tools in the world won’t help if you don’t use them wisely. Test tools might make it very easy to specify tests, but whether you’re specifying the right tests at the right time is up to you. Lisa’s team found that too much detail up front clouded the big picture to such a degree that the programmers didn’t know what to code. This won’t be true for every team, and at some point we do need details. The latest time to provide them is when a programmer picks up a coding task card and starts working on a story.

Writing detailed test cases that communicate desired behavior effectively requires both art and science. Poorly expressed scenarios and poorly designed test cases can create more confusion than they resolve. Experiment so that you can find the right level of detail and the right test design for each story. Let’s look at some strategies to help you use tools successfully to write useful business-facing tests.

Build Tests Incrementally

After we have defined our high-level acceptance tests so that the programmer knows what to start coding, we can start elaborating on the rest of the story tests. We can work closely with the programmer to ensure we automate the best possible way.

When a programmer starts working on the programming tasks for a story, start writing detailed tests. For those of us who enjoy testing, it’s tempting to go for the biggest “smells” right away, the areas where we think the code might be fragile. Resist the temptation. Make sure the most obvious use case is working first. Write a simple, happy path automated test to show the code accomplishes the most basic task that it should. After that test passes, you can start getting more creative. Writing the business-facing tests is an iterative process.

Lisa’s Story

I start writing executable business-facing tests that support the team by writing a simple FitNesse test based on examples that the product owner provides. I show this to the programmer working on the code. He can make suggestions for changes right then, or he might modify the test himself as appropriate when he’s ready to automate it. Discussing the test often leads the programmer to realize he missed or misunderstood a requirement. We might need another three-way conversation with the customer. The programmer updates the code accordingly. We can also show the test to the product owner to make sure we captured the behavior correctly.

After the simple test passes, I write more tests, covering more business rules. I write some more complex tests, run them, and the programmer updates the code or tests as needed. The story is filling out to deliver all of the desired value.

—Lisa

Chapter 18, “Coding and Testing,” goes into more detail about how testers and programmers work together to test and code.

Confine each test to one business rule or condition. At some point you can automate or manually perform more complex scenarios, but start by covering each condition with a simple test. If you’ve followed our recommended thin slice or steel thread pattern, the first set of tests should prove the first thin slice end-to-end. As your automated tests pass, add them to the regression suite that runs in a frequent build process.

Keep the Tests Passing

After a test passes, it shouldn’t fail unless the requirements were changed. If that happens, the test should be updated before the code is altered. Of course, if a test was forgotten as part of a requirement change, we expect it to fail. It did its job as change detector. At this time, the test will likely need to change to get it passing.

Whenever a test fails in a continuous integration and build process, the team’s highest priority (other than a critical production problem) should be to get the build passing again. Don’t comment out the failing test and fix it later; that’s the road to perdition. Soon you’ll have dozens of commented-out tests and a lot of technical debt. Everyone on the team should stop what they’re doing and make sure the build goes “green” again. Determine if a bug has been introduced, or if the test simply needs to be updated to accommodate intentionally changed behavior. Fix the problem, check it in, and make sure all of the tests pass.

Lisa’s Story

Early on in our agile efforts, my team wasn’t fixing broken tests fast enough. I wrote “Tests are not temporary!” on the whiteboard to remind everyone that once a test passes, it needs to keep passing. A few days later, the words “but testers are!” had been added to get back at me. We did get much better at keeping our builds “green” after that.

—Lisa

One passing test leads to another. Keep your tests current and maintainable with refactoring. Extend them to cover other test cases. The various combinations and scenarios might or might not become part of the regression suite after they pass. We want our regression suite to run in a timely manner, and having too many tests for edge cases would slow it down.

Use Appropriate Test Design Patterns

When designing tests, look at different patterns and choose the ones that work for you. Keep them as simple as you can. Before you can design tests, you have to identify the ones you need. Pierre Veragen coined the term test genesis patterns to note the patterns that help you think of tests. Examples and use cases feed into our test genesis patterns.

Build/Operate/Check

Lisa’s team often goes with a build/operate/check pattern: Build the input data, in memory or actually in the database, depending on the purpose of the test; invoke the production code to operate on those inputs; and check the results of that operation. Some teams call this setup/execute/validate. For example, to test the invoice presented to a new account holder, set up the fees to be charged, input the properties of the account that relate to fee amounts, run the code that calculates the fees, and then check to see what fees were actually charged. See Figure 9-9 for an example of a test that sets up a loan with a specified amount, interest rate, term, payment frequency, and service start date and then checks the resulting amortization schedule. The test data is built in memory, which makes for a speedy test. A “teardown” fixture (not shown) removes the test data from memory so it won’t interfere with subsequent tests.

Figure 9-9 Example test with build/operate/check pattern

If there’s a need to test the application’s data access layer, tests can run using an actual database. Each test can insert the test data it needs, operate on it, check results, and delete the data. Testing with data in a real database can be a means of automating a test against legacy code whose data access and business logic layers aren’t easily separated.

Notice that the “check” table in the example uses a declarative style, with each row forming an independent test case, without changing the state of the system. Each row in our example tests a line in the loan amortization schedule. In the next section, we’ll look at patterns that are in a procedural style, with steps that change or test the state of the system.

Time-Based, Activity, and Event Patterns

Sometimes a timeline-based procedural pattern reflects the business better. For example, when testing a loan, we want to make sure interest and principal are applied correctly for each payment. The amount of interest depends on the date the payment was received and the date of the last payment processed. We want a test that simulates taking out a loan for a certain dollar amount, interest rate, and time period, and then over time simulates the borrower sending in payments, which are received and processed. Figure 9-10 shows a simple example of a FitLibrary “DoFixture” test that takes out a loan, checks the payment amount, posts the borrower’s payments, receives the payments and processes them, and then checks the interest, principal, and loan balance amount. It also checks the loan default state.

Figure 9-10 Sample time-based test

Depending on the domain, a time- or event-based approach might simulate the actual business processes better and be more understandable to business experts than a declarative type test. Other customers might find the declarative table style simpler to understand, because it hides the procedural details. Different patterns work best for different situations, so experiment with them.

Learning More

Your team should educate itself on test patterns that help drive programming. Finding the right pattern for each type of test ensures the test communicates clearly, is easy to maintain, and runs in an optimal amount of time. See the bibliography for more invaluable resources on test design, such as Gerard Meszaros’s xUnit Test Patterns: Refactoring Test Code.

Bring programmers and testers together to brainstorm test approaches and to help decide what tests can be automated and how the code should be designed to support testing. Business logic and algorithms should be accessible by test fixtures, without having to go through a user interface or batch scheduling process. This enables test-driven development, which in turn produces testable architecture.

A common approach to automating tests is by driving tests with keywords or action words. This can be used with tools such as Fit and FitNesse, or Ruby with Watir. We’ll explain this next.

Keyword and Data-Driven Tests

Data-driven testing is a tool that can help reduce test maintenance and enable you to share your test automation with manual testers. There are many times when you want to run the same test code over and over, repeating only the inputs and expected results. Spreadsheets or tables, such as those supported by Fit, are excellent ways to specify inputs. The test fixture, method, or script can loop through each data value one at a time, matching expected results to actual results. By using data-driven tests, you are actually using examples to show what the application is supposed to do.

Keyword-driven testing is another tool used in automated testing, where predefined keywords are used to define actions. These actions correspond to a process related to the application. It is the first step in creating a domain testing language. These keywords (or action words) represent a very simple specification language that non-programmers can use to develop automated tests. You still need programmers or technical automation specialists to implement the fixtures that the action words act on. If these keywords are extended to emulate the domain language, customers and nontechnical testers can specify tests that map to the workflow more easily.

The sample spreadsheet in Figure 9-11 shows how one company used action words to automate their test setup. The same action words can be used to test. The words Signup, Signoff, and CCDeposit are words that are domain-specific. Their users could easily write tests without understanding the underlying code.

Figure 9-11 Sample test spreadsheet with action words

Combining data-driven and keyword-driven testing techniques can be very powerful. Fit and FitNesse use both keywords and data to drive tests. The other tools we’ve described in this chapter can also accommodate this approach.

Any test strategy can run into trouble if the code isn’t designed to be easily tested. Let’s take a look at testability concerns.

Testability

Business-facing tests built with appropriate design patterns and written ahead of any coding help the team achieve a testable code design. The programmers start by looking at the business-facing tests, perhaps together with a tester, analyst, or customer, so that the need to execute those tests are always in their minds as they proceed with their test-driven design. They can build so that the tests provide inputs and control run-time conditions.

Janet’s Story

I ran into a snag when I was trying to automate some GUI workflow with Ruby and Watir. The calendar pop-up feature was not recognized, and the data field was read-only. I took my problem to one of the programmers. We paired together so that he could see the issue I was having. The first thing he did was to understand the calendar feature. He thought it would be too difficult to automate the test, so he suggested another alternative. He created a new method that would “fool” the input field so it would accept a date into the text field. We knew the risk was no automation on the calendar, but for simplicity’s sake we went with his option.

Not all code is testable using automation, but work with the programmers to find alternative solutions to your problems.

—Janet

Let’s look at techniques that promote design of testable code.

Code Design and Test Design

In Chapter 7, “Technology-Facing Tests that Support the Team,” we explained how test-driven development at the unit level ensures a testable architecture. This is true for business-facing tests as well. The layered architecture Lisa’s team designed works just as well for functional testing. Testing can be done directly against the business logic without involving the user interface, and if appropriate, without involving the database layer. This doesn’t mean that the database layer doesn’t need to be tested. It still needs to be tested, just maybe somewhere else.

Testability has to be considered when coding the presentation layer as well. GUI test tools work better on well-designed code developed with good practices.

Lisa’s Story

When I first started trying to automate GUI tests using Canoo WebTest, I discovered that the HTML and JavaScript used in the system didn’t comply with standards and contained many errors. WebTest and the tool it’s built on, HtmlUnit, required correct, standard HTML and Javascript. Specifying tests depended on good HTML practices such as giving each element a unique ID. The programmers started writing HTML and JavaScript (and later, Ajax) with the test tool in mind, making test automation much easier. They also started validating their HTML and making sure it was up to industry standards. This also reduced the possibility of the application having problems in different browsers and browser versions.

—Lisa

Coding and testing are part of one process in agile development. Code design and test design are complementary and interdependent. It’s a chicken-andegg scenario: You can’t write tests without a testable code design, and you can’t write code without well-designed tests that clearly communicate requirements and are compatible with the system architecture. This is why we always consider coding and testing together. When we estimate stories, we include time for both coding and testing, and when we plan each iteration and story, we budget time to design both tests and code. If automating a test proves difficult, evaluate the code design. If programmers are writing code that doesn’t match customer expectations, the problem might be poorly designed tests.

Automated vs. Manual Quadrant 2 Tests

We’ve assumed that at least a good-sized portion of the tests that guide programming will be automated. Manual test scenarios can also drive programming if you share them with the programmers early. The earlier you turn them into automated tests, the faster you will realize the benefit. Most manual tests fall more into the “critique product” quadrant where we might learn things about the story we hadn’t anticipated with the initial set of tests.

In Part IV, “Test Automation,” we’ll dive into developing a successful test automation strategy and look at considerations such as building your own tools versus using third-party or open source tools.

That doesn’t stop us from writing tests that might not be appropriate for automation. Don’t sweat the details when you’re writing tests. You might come up with one-off tests that are important to do but not important to repeat over and over in a regression suite. You might start thinking about end-to-end scenarios or springboards to exploratory test sessions that might be facilitated with some automation but need an intelligent human to conduct them in full. You’ll figure that out later. Right now, we want to make sure we capture the customer’s critical requirements.

Start with a simple approach, see how it works, and build on it. The important thing is to get going writing business-facing tests to support the team as you develop your product.

Test Management

If we’re automating tests, it makes sense to present them in the automation tool framework, even if they’re not yet executable. We want some way for all tests, even those that won’t be automated, to be accessible to everyone on the development team and understandable to our customers. There are lots of options available that let everyone on the team see tests. Wikis are a common way to share test cases, and some tools such as FitNesse use a wiki or similar tool, enabling narrative requirements, examples, and executable tests to coexist in one place.

Chapter 14, “An Agile Test Automation Strategy,” goes into more detail on how to manage automated tests.

Tests should be included in your source code control, so that you can track which versions of the tests go with which versions of the code. At the very least, have some kind of version control for your tests. Some teams use test management tools or comprehensive test frameworks that might integrate with requirements management, defect tracking, or other components.

Summary

In this chapter, we’ve looked at tools you might want in your toolkit to help create business-facing tests that help drive development and guidelines to make sure the tools help rather than get in the way. The tools and guidelines included the following:

Teams need the right tools to elicit requirements and examples, from the big picture down to details, including checklists, mind maps, spreadsheets, mock-ups, flow diagrams, and various software-based tools.

Tools to express examples and automate tests, below and through the GUI, are also essential to agile test automation. Some of these tools include unit test tools, behavior-driven development tools, FitNesse, Ruby with Watir, Selenium, and Canoo WebTest.

“Home brewed” test automation helps teams keep the total cost of ownership of their automated tests low.

Driving development with business-facing tests is one way agile teams are motivated to design testable code.

Test strategies for building your automation should include building your tests incrementally and making sure they always pass. Design patterns can be used to help you create effective tests.

Keyword and data-driven testing is a common approach that works with the tools we’ve discussed in this chapter.

Consider testability in your code design, and choose your test tools wisely, because they need to work with your code.

We need some way to organize tests so that they can be used effectively and put into version control.