Criteria for Completion of Testing - Testing Tools

Surprisingly, it is very difficult to say when the testing phase is complete and when the product can be released. As you keep testing the software, it may happen that you keep getting a defect—and this process may go on forever. On the other hand, you may not detect any defect, but what is the guarantee that there is no hidden defect? And, that defect may show its effect on the day the software is delivered to the customer. See the irony, after all the effort of developing and testing the software, you are not sure whether it is ready for shipment! This proves our point once more: "testing is difficult". Instead of relying on gut feeling, we need to arrive at a scientific method to declare that the testing is complete and the product is ready for shipment.

When is Testing Complete?

It is very difficult to say when testing is complete. Three criteria used in practice for completion of testing are:

  • When you run out of time.
  • When you run out of money.
  • Based on statistical criteria.

In practice, unfortunately, the first two criteria are followed. During the planning stage, certain time and money (or effort) are allocated for the testing process. The test team keeps testing the software, and when they run out of time (or money), the product is delivered (to avoid the penalty for late delivery). Most managers do not realize that this is a very dangerous practice because if the software fails at customer's site, the reputation of the organization is at stake.

A more practical approach for declaring that the testing is complete is to use the third criterion. After the coding is completed, and testing begins, initially many defects are detected. Slowly, the number of defects found in a given time (say, in a day or week) keeps reducing. The graph showing the number of defects found every week will be as shown in Figure If the number of defects found per week remains less than a pre­defined threshold consecutively for three weeks, then the software can be considered a mature product and released to the client.

'Completion of Testing' based on defects Found per Week

'Completion of Testing' based on defects Found per Week

Classification of Defects

All defects are not of the same type. Some defects may result in system crash. Some defects may be very minor (such as a spelling mistake in an error message). So, the defects need to be classified based on its impact on the functionality of the software. The defects can be classified as

  • Critical
  • Major
  • Minor

Critical defects result in a system crash, or the application may not be useable at all if this type of defect occurs. In a time critical application, if the timing is not within the limits or if the response time is not within limits; then also the defect can be termed as critical. Inconsistent performance of the application is also a critical defect.

Major defects will not lead to a system crash, but may result in some portions of the application difficult to use, such as difficult navigation of the menus.

Minor defects can be tolerated, such as lack of help for some functionality, a spelling mistake in an error message, incorrect ordering of control buttons in a GUI etc.

While recording the defects in a defect log sheet, the severity of the defects also needs to be noted down. If a critical defect is present in the code, then the software is not ready for delivery. Only a few major defects are allowed. Similarly, a threshold needs to be kept on the number of minor defects. As the threshold depends on the application, the project manager/test team member has to use his judgment in fixing the thresholds on major and minor defects.

After analyzing the test report, the project manager needs to decide whether the software is ready for delivery or not.

All rights reserved © 2020 Wisdom IT Services India Pvt. Ltd Protection Status

Testing Tools Topics