Lessons from developing agile team Agile Testing

This section describes lessons learned on the agile testing project in the following areas:

  • developer relations,
  • a project is a project
  • how to decide when to release
  • definitive evidence,and
  • the epilogue.

Developer Relations

We had enthusiastic buy-in from all parties when we started the project;in principle the approach was well received,though there was some initial skepticism about whether it would work in practice,and some surprise when the results promised were delivered. However, in one particular area we experienced significant pushback; our relationship with the developers started to break down when the bugs began to be found!

As is the case in many parts of the finance sector, staff are paid bonuses based on their ability to deliver high-quality solutions into production.There was an expectation that we would not find much wrong with the system and that our efforts were going to be a bit of a sideshow. However, we found so many bugs that the first release was two months late going into production.

In some organizations there is a significant “blame culture. ” The testing process shone a spotlight on individuals when they were developing their code.So from the developers’perspective, the test team became “the enemy.”

The situation became worse; as the developers gained an understanding of how the testing process worked,and by introducing subtle changes to the code each time the system changed, they were able to make the test scripts fail.We did not realise at he time but subsequently learned that there was a deliberate initiative to make changes to the solution user interfaces in order to disrupt the testing process so that the testers would be perceived as the obstruction in the development process.

In order to address the issue of poor developer relations,in all future projects of this nature we followed the following approach:

  • Identify facts and do not speculate on the causes– having identified the facts, let the developers do the investigation so that they can become heroes for having found the cause of the problem and,as a minimum, have time to build a story for their managers.
  • Ensure that the design approach takes into account all aspects of the deliverable
    such as changes to the code, changes to the regression test harnesses,preparation of test data. In this way,we estimate and optimize the time taken to perform all these activities to complete a change. Too often we found design decisions were taken to minimize the development process at the expense of the testing activity and to the detriment of the overall delivery. If a developer has the choice of a three-day work package that incurs ten days of testing effort as opposed to a five-day work package that incurs two days of testing effort, too many times the developers chose the former. The approach needs to motivate best behavior for the overall team.
  • Ensure that test specifications are included in the work package–in order to define the completion criteria and to ensure the work package is complete when the test criteria are satisfied.Any subsequent bugs are not the developer’s responsibility but rather that of the person who wrote the test specification. Thus, the developer has a structured approach to completion and does not waste time in random testing“in the hope” of finding a bug.Problems will arise later inevitably, but they will be integration problems arising from issues that had not been predicted.
  • Developing the previous points,regression testing cannot remain separated from the development process or it is doomed to failure in the medium term.Use of the tools and maintenance of the test harnesses have to be undertaken by the developer and changes in the code and in the regression tests associated with that code made as part of the same work package or else the regression tests will not be maintained.
  • Management responsibility for development and testing must be delegated to as low a level as possible– the test manager should be responsible for thedefinition of best practice,for quality standards,and for owning the regression testenvironment in order to ensure that it is being run every night so that new bugsare isolated as quickly as possible and passed back to the developers.However,the developers must maintain both the code and the associated test harness.
  • As a consequence of the best practices set by the test manager,we can develop an integrated set of test components that linked together support system testingbut can be decomposed into elements that can support both integration testingand module testing.This enabled us to shorten development cycles considerablyby reducing the time for development of individual modules,the time taken fortesting, and the number of bugs detected at a late stage.

A Project Is a Project

A fundamental aspect of our approach in managing the testing was to view testing activities as “just another project,” and as such,to ensure we followed good project management best practice,such as that presented in PRINCE2[22].This approach included

  • identifying the project sponsor,agreeing terms of references,and confining yourself to those terms of reference.You cannot expect to solve all the problems theorganization is experiencing,so focus on solving those you are asked to address and ensure that you retain the confidence of the project sponsor by reporting onprogress against plan for the agreed objectives;
  • identifying all the people affected by the project and ensuring that there is a mechanism for keeping them apprised of developments and progress;
  • creating and following a project plan,identifying the tasks to be undertaken by all participants,and ensuring that they are tasked through the project steering committee to deliver against the plan;
  • identifying early when there is risk that an objective may not be delivered by the client’s staffand giving the project sponsor sufficient warning so that there is areasonable chance of the situation being resolved;
  • budget for testing a “rate of run project” and accept that there will be a regular monthly cost regardless of the level of work.Once the initial test tools areavailable,running through the regression test process is highly dependent on the delivery by the developers;so one is a hostage to fortune if a testing organization undertakes work on a fixed-price basis. However,if you have testers who are also developers,you can consume time to benefit by getting testers to undertake some development-related tasks;and
  • ensuring that when your role as a consultant is completed,then you exit.Define the exit criteria,meet them,and go! We are all passionate about testing,and it needs to be a permanent function,so help the customer to recruit a member of staff to continue the good work using the framework you have put in place.

How to Decide When to Release

It was expected that our approach would find problems;the challenge we faced was how to report them and how to prepare senior management to deal with them.

In our case,many problems once discovered were considered to be so serious that they had to be fixed immediately,and the new release they were associated with could not be delivered until they were resolved.

This was the single most significant factor in delaying the releases.In retrospect,we should have worked harder to create a regime where if the release was at a state where it was an improvement and there were no new known problems being introduced,then we should have proceeded with the release anyway on the basis that we were at least improving the quality of the code.

An alternate approach,and one that would have managed the expectations of the stakeholders more effectively,would have involved us making it clear that there was a significant probability that we would find so many issues,that the overall testing effort was likely to take at least three months,and perhaps up to six months.This would also have allowed the stakeholders to change their overall approach to the project–perhaps taking a different view on how to proceed.

As a result,even though the agile test approach was finding defects,getting them fixed,and improving the quality of the software,we were seen as the cause of the late release,rather than as the agents of improvement.

Definitive Evidence

It was found that using software tools allowed the creation of highly valuable reports.Experience showed that when the situation is extremely serious you can nevertheless create hope by performing analysis using heat-maps3 and other categorization reports to identify where the trouble areas are and what the trend lines are.

These types of reports provide definitive evidence to enable senior managers to get involved in solving the problem by allowing them to determine where the problem is,and to take action based on hard evidence created from metrics.Producing trend graphs shows that,even in a dire situation,once the trend line starts to improve,the management action taken can be demonstrated to be having a positive effect.


Using the lessons learned from this project,we undertook a number of similar projects for other major financial institutions. The benefits of implementing this successful agile approach for these other customers include

  • improving the functional quality of the delivered software;
  • improving system robustness and availability;
  • verifying that the performance of the software was capable of satisfying specific throughput levels(through rigorous performance testing);
  • establishing the upper limits of scalability to support business planning,so that system hardware upgrades could be planned;and
  • identifying,isolating,and rewriting performance bottlenecks to improve performance and scalability.The agile testing approach described in this case study continues to be a useful and effective solution that will continue to be used and refined in future projects.

All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd DMCA.com Protection Status

Agile Testing Topics