Agile Automation Agile Testing

This section provides best practice guidance on the use of automated software tool support for agile methods. Automation has the potential to save significant time, effort, and cost on agile projects, but, where selected or used poorly, it can also cost a project time, effort, and money!

As Jon Tilt puts it, “There is always the temptation to architect yet another automation framework at the start of the project. Don’t! Look around at the simple tools that already exist and build on them as required. Developers and testers are much more likely to use something simple that is quick and easy to use.”

In addition to quality management and testing products, this section discusses agile best practices relating to other tools such as requirements management, configuration management, build management, and process enactment tools.

Select a Tool in Haste, Repent at Leisure

Phillips reports that the agile project he worked on was forced to change tools several times during the course of the project; the time and effort needed to select a replacement tool, plus the nugatory effort expended on gaining familiarity with the previous tool, and wasted effort using it, can all jeopardize the success of an agile project, where the goal is typically to reduce the effort and cost of delivering the product.

Prior to starting an agile project there are a number of things you can do to try to avoid making poor automation decisions:

  • Determine whether there are company standards in place for tool use(but don’t follow these slavishly if they are clearly wrong for your project).
  • Discuss the need for tool support with knowledgeable colleagues and consider the wider networking opportunities of attending appropriate special interest group events.
  • Be aware of the need for a particular product to integrate with other tools(either in place or planned to be acquired).
  • If the project is large enough, of sufficiently long duration, and with available resources, consider conducting a formal evaluation of tools(particularly if they are products you will need to purchase). A tried and trusted tool evaluation scheme that can be used to ensure formality in any evaluation you undertake is given.

Static Analysis Tools

These products typically allow the source code to be inspected to highlight quality issues without executing the code(some are even customizable to allow company coding standards and house coding style issues to be detected). Arguably, these relatively simple but highly effective tools should be another tool in the developers’ toolbox and should be used on a frequent basis to check recently written or modified code.

As with many of the products discussed in this section, static analysis tools can be particularly effective when used in conjunction with an automated build and testing approach. For example, Evansdescribes a system that combines continuous integration and build management tools to automate the creation of a new build of the software. During this process, automated unit, functional, and static analysis tests are run against the code to determine whether the build is a success. Should the build fail due to defects being identified, the build s rejected, and the defects are corrected before rerunning the build process once more.

Automated Unit Test

The use of automated unit test tools(such as [50]) appears to be an agile practice adopted in almost every case study (Cassidy, Wilson, Kingston, Warden, Tilt, and Stapp and Nowakowska). These simple and effective tools allow unit tests to be compiled in concert with code development and executed to ensure the code meets its requirements and performs as expected.

Although a wide range of unit test products are available to purchase or download as open-source tools, many developers also write their own unit test tools. Whatever solution is adopted, you should take into account support issues, the availability of practitioners who are familiar with the tools, and the need to maintain and update these tools.

Combined with test harness tools (used where the unit under test needs to interact with components that have not yet been coded)such as mock objects(Cassidy, for example), automated unit tests provide an inexpensive and effective means of initially testing a component as well as continuing to test the successful functioning of the component following a new build, release, or other changes to the software(such as code refactoring).

Finally, employed in concert with continuous integration and automated build tools, automated unit tests become a highly effective solution to accelerate development through continuous code quality assessment.

Test Harness Tools

It is often the case, particularly in the early stages of development, that code will be written that needs to interact with other components that may not have been developed at that point in time. Under such circumstances a typical solution is to employ a test harness to simulate the operation of the missing component(s).

It is not unusual for developers to spend some of their time writing and testingtheir own test harnesses(or reusing those generated by colleagues). Although of value in supporting more efficient unit testing, creating these test harnesses inevitably expends time, effort, and expense in generating a temporary tool that almost certainly will not have any purpose once the current project is completed.

To inctivity of developers and to save them the time and effort of building their own test harnesses, a range of open-source test harness tools have been developed that are available for use in unit testing. Evans, for example, describes his experiences in using the so-called mock framework test harnesses and their role in automated build and test, while Cassidy describes how his project began by using a number of freely available mock frameworks(such as EasyMock and then went on to develop its own generic reusable mock framework–SevenMock.

Functional Testing Tools

Functional testing tools, such as the classic capture–replay tools, allow testers to capture user navigation against the application under test and any specific verifications(that a particular window has appeared, or some specific calculation has been made correctly, for example)in the form of an automated script. This script can be replayed against later builds or releases of the software to ensure the functionality has not changed and that the new changes have not introduced any bugs.

The benefits of the use of such tools on an agile project are clear:

  • Automated test scripts can be scheduled to run overnight or over the weekend, allowing additional unattended testing to be performed, increasing confidence in the software.
  • The more often you reuse the automated scripts to perform regression testing of new builds and releases, the more time, effort, and cost you save the project.
  • Unless you are dealing with an extremely large suite of automated test scripts and are facing very long test suite execution times, you do not have to take the risk of having to pick and choose which scripts to execute–you simply run them all.
  • Automated capture–replay tools do not get tired, do not become bored or careless, and will continue to replay the test scripts and report on the results in a thorough and repeatable manner.

A number of case studies report significant benefits from the adoption and use of capture–replay tools for functional and regression testing(such as Denning, Cassidy, Phillips, and Tilt,)and variously cite reductions in testing time, effort, and cost, as well as improvements in quality through their use.

However, such tools are not a universal panacea, and a number of case studies report issues with using such tools.Knowles, for example, reports that well-managed manual testing delivered excellent results on the migration project described in his case study. In this instance, capture–replay tools were rejected on the grounds that the solution was too expensive to maintain in a rapidly changing development environment, with high cost and effort involved in maintaining the scripts. Phillips cautions against underestimating the time and effort required to develop and maintain a test automation infrastructure and particularly one where the capture–replay tool has to be integrated with a number of other products.

The characteristics of those projects that seem to be particularly suited to the application of capture–replay tools are as follows:

  • A stable user interface(Phillips)should be available–the screens in the application under test should not be in a constant state of change(such as that caused by user interface requirements churn). Under such circumstances, the cost of maintaining the automated test scripts may well outweigh the cost of manualtesting.
  • There should be numerous and frequent builds and releases of the applicationunder test. Under such circumstances, thorough and extensive manual testing becomes unrealistic, and an automated solution is much more appropriate(Tilt). A further benefit of frequent builds and releases is that each time the automated test suite is reused, the overall return on investment in the tool is improved.
  • The application under test is complex, involving numerous screens and complicated navigation and with a large amount of functionality to test. Where the software is relatively simple, more cost-effective manual testing may suffice.

As a final thought, if you combine the introduction of capture–replay testing tools with your metrics program, you will be able to work out the precise benefits obtained from using the tool and make an informed decision on whether it is of value to your project.

Requirements Management Tools

A number of the case studies report the benefits of using tools to document and maintain requirements(Kingston, and Chana, for example).

From a quality management perspective, the ability to provide up-to-date requirements information to test managers and test analysts for test planning and resourcing, and test case design purposes were commonly cited uses of such tools. Kingston makes a strong case for the use of such tools to demonstrate formal test coverage against requirements, while Chana describes the value of the change history supported by such tools in providing a useful source of traceability(allowing the persons who requested, reviewed, and authorized a particular change to the code base to be easily identified, for example).

Kingston makes a further point about the utility of such tools on agile projects where not all the stakeholders can be co-located; the ability for remote stakeholders to access the requirements tool via a web-based interface enabled the offshore staff to work more closely with other members of the team and allowed all stakeholders(irrespective of where they were based) to contribute to the requirements elicitation and documentation process.

Sewell and Hodgkinson both describe the value of automated support in requirements elicitation, specifically in documenting and analyzing use cases within the EssUP and RUP frameworks, respectively.

Build Management Tools

A large number of the case studies document the benefits of adopting a continuous build or integration approach(Gilb, Thomas, Knowles, Wilson, and Evans,).

Unless the project is of trivial size and complexity, a build management tool must be considered for use in starting and running the build process. Many of the case studies(such as Evans; Phillips, Wilson, and Tilt,)also report the benefits of integrating other automated tools such as unit test, functional test, and code analysis tools) with the build management tool to ensure that those builds that contain defects are rejected.

Configuration and Change Management Tools

Configuration and change management are well-established best practices in both traditional and agile software development projects, and many of the case studies report the value of using them(Thomas, and Cassidy, for example).

Numerous configuration and change management tools are available, including commercial products, open-source tools, and those developed in-house for example), that can be employed in agile projects. In selecting such a product, you should be aware of support issues, the availability of practitioners who are familiar with the tools, and the need to maintain and update these tools.

Finally, be aware of the need for the configuration and change management tools you select to integrate with other tools that you are already using on a project or those that are planned to be acquired.

Process Enactment Tools

A number of the case studies emphasize the importance of providing good process guidance to the stakeholders involved in developing and testing the software(Sewell, Kingston, and Hodgkinson, for example); if you want your agile project to be successful, it is essential that the practitioners working on the project understand and are able to employ agile best practices.

Several of the case studies also provide cautionary tales about projects that claimed to be following an agile approach, but which in reality were not using agile best practices(Thompson, Allott, May, for example). The reasons for this can be diverse:

  • Thompson suggests in his case study that a company made claims about their agile credentials to win a development contract, but that in practice there was very little evidence that the company had used an agile approach.
  • Allott suggests that an agile project needs particularly well-trained and experienced agile practitioners if it is to be successful, otherwise weak or poorly trained staff will fall back on
    more familiar(nonagile)practices.
  • Warden highlights a litany of examples where an agile approach was shoehorned into a project, but where true agile practices were not actually adopted(such as the failure to ensure that the end users performed the sprint QA/sign-off, instead of this step being performed by the analysts who wrote the specifications).
  • Similarly, May documents a number of departures from agile(in this case Scrum) practices, which jeopardized the success of the project he was involved in, including poor user stories, interrupted sprints, excessively long sprints(greater than six weeks), failure to co-locate stakeholders, and poorly focused stand-up meetings of too long a duration.

There are a number of options for addressing these issues:

  • Ensure that you employ experienced agile staff who have a good track record of using agile practices successfully.
  • Provide staff with appropriate training on the agile method being used and its practices before you begin the agile project.
  • Provide easy access to agile best practice guidance–such as practice sheets(Sewell)or easily accessible online process guidance. Another possible option that has become available recently is to deploy a socalled process enactment tool for example). Such tools provide a rich integrated development environment in which the use of agile best practices is enforced by the tool itself. For example, such tools can
  • allow the addition of new team members to a project using standard templates for roles and responsibilities, with defined lines of reporting, privileges, and permissions;
  • provide defined workflows within which tasks can be assigned to other team members, which can then be accepted and worked on, with the developer entering estimates of the time and effort required to complete them as well as the ability to update the task with the actual values;
  • provide real-time reports on individual and project progress and generate accurate estimates, such as the timescales required to complete a particular task;and
  • be customized to match the specific agile practices of a particular agile method. For example, MacDonah describes the implementation of the EssUP process in a commercial process enactment product.

All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd Protection Status

Agile Testing Topics