Functional Testing Tools Workshop: Retrospective and Envisioning the Next Generation


Elisabeth Hendrickson and Joe Wilk

Program Approved Date

February 2007


Automated functional testing is an integral and essential part of Agile development. For example, on XP and many other Agile projects, functional tests codify the system requirements. As requirements specifications, functional tests must be readable: clear, succinct, and expressed in the language of the business domain. As an automated safety net, the tests must be maintainable: built with reusable domain specific testing language components, easy to change as the requirements change.

Tool support for automated functional tests has grown significantly in recent years. There is a large variety of testing tools/frameworks available commercially and as open source. The FIT framework was a significant boost to the state of the art of automated functional testing, both in terms of the syntax of the specification (tables), the detailed test execution feedback (cell by cell), and the development/execution environment (desktop tools rather than development or specialized tools).

It’s time for another significant boost to the state of the art.

  • We are lacking integrated development environments that facilitate things like: refactoring test elements, command completion, incremental syntax validation (based on the domain specific test language), keyboard navigation into the supporting framework code, debugging, etc.
  • We need more expressive test specification languages, possibly integrating executable: text, tables, shapes, and colors together into a single test.
  • We need specification languages that can describe user interaction in a readable and maintainable fashion.
  • We need to be able to view/navigate the tests in multiple different ways in order to see how the pieces of the puzzle contribute to the bigger picture of the domain/feature: organize tests based on their domain context; search for tests based on user-defined keywords (cross cutting concerns).
  • ...and things that we haven’t even thought of that will take us out of the current box, and into a new level of effectiveness.

Main Activities

Envisioning possibilities for state of the art

Fund and organize workshop(s) for envisioning the state of the art of automated functional testing. The outcome of the workshops would be:

  • A comparison of current approaches (such as table/keyword driven v. domain specific languages)
  • A catalog of existing open source tools
  • A set of requirements that define what makes a given testing tool appropriate for an Agile project
  • Vision(s) for the next generation

An example of an existing workshop we might use as a model is AWTA, the Austin Workshop on Test Automation, hosted by Bret Pettichord once a year. Past AWTAs have resulted in increased activity of the Watir library for web testing and the Selenium testing framework.

Build Awareness and Community

We seek to fund efforts that result in high-quality tool-related content to be published on the Agile Alliance web site, such as:

  • Tool catalogues
  • Tool tutorials
  • Other tool-related educational materials

We will also strive to build relationships with other organizations:

  • Contact Apache, Eclipse, etc to explore whether they have any relevant projects already under way
  • Create a Google group and have online discussions on the topic prior to the workshop

Anticipated Questions

How Will the Program Advance the Mission of the Agile Alliance?

This program will “Help Agile Teams Perform”, by bringing them tools that they can use to more effectively create automated functional tests. The intent of the program is to deliver concrete value to a large number of software teams within the term of the program (2007 – 2008).

What Are the Commercial Aspects of the Program?

Content and tools developed under this program would be made freely available as open source or through the Creative Commons license.We will not seek to create tools for sale, nor will we directly support commercial development efforts