In this article, I would like to look at a very important kind of metric—one that will allow you to evaluate the quality of a project and define steps to improve it. If applied correctly, these metrics can help address the most important testing artifacts—defects.

1. Defect Density

Defect density is the share of defects attributable to a particular module during a sprint or release. The formula for Defect Density is:

Defect Density =  Number of Defects / Size of the Software

For example, if a model has 1000 lines of code and 10 defects are found, the defect density would be 10 / 1000 = 0.01 defects per line of code.

The purpose of this metric is to highlight which part of the software is the most problematic. This information will help when evaluating and planning work with a given module, as well as when analyzing risks. The reasons for a large number of defects in one particular module may be due to several different causes: poor quality requirements, developer qualification, technical complexity, etc. In any case, this metric will immediately draw your attention to the problem area.

2. Percentage of Critical Defects

It’s a measure indicating the proportion of reported defects classified as critical out of the total number of defects. Critical defects are typically those that have a severe impact on the functionality of the software or pose a significant risk to the project.

A higher percentage of critical defects may indicate a higher level of risk and severity in the software. It may trigger additional reviews, testing, or mitigation strategies to address critical issues promptly. However, it’s important to note that the definition of “critical” may vary based on the project and industry. Thus, teams need a clear definition of severity levels to categorize defects consistently.

3. Regression Ratio

The purpose of this metric is to show where the team’s efforts are spent: do you spend more time creating and debugging new features or do you spend most of your time patching existing parts of the software?

Regression Ratio = Number of New Defects / Number of Fixed Defects

The closer the ratio is to 0, the fewer errors were introduced into the existing functionality when implementing new requirements. If the value is more than 0.5, then you spend more than half of your time on restoring previously working software functions.

4. Recurring Defects

This evaluates the quality of development and defect correction, as well as the complexity of the product or a single module. The metric can be calculated for the whole software product, a separate module, or a single functionality.

Recurring Defects = Number of Recurring Defects / Total Number of Defects

The closer the obtained value is to 0, the less old bugs are repeated during development. If the coefficient is more than 0.3, it may indicate either the technical complexity of a module and high interconnectedness of requirements in it, a buggy architecture, or the fact that the fixes are being done poorly.

5. Defects in Code of a Specific Developer

This metric provides insights into the quality of code produced by individual developers and can help identify areas for improvement or additional support. It constitutes the total count of defects discovered in the code written by a specific developer.

A higher number of defects may suggest that the developer’s code requires additional scrutiny or improvements. It can be an indicator of the effectiveness of the developer’s testing or the team’s code review practices. Identifying developers who consistently contribute code with a higher number of defects can inform targeted training or mentoring efforts.

The defects must be properly attributed to the responsible developer through an effective tracking and management system. Besides, the definition of a “defect” and the criteria for attribution should be clearly defined to ensure consistency.

This metric should be used judiciously and with consideration for various factors. Some developers may work on more complex features or areas of the codebase, naturally leading to a higher likelihood of defects. It’s important to encourage a culture where the goal is not to attribute blame, but to identify areas for improvement and learning.

6. Average Defect Lifetime

This metric measures the average duration it takes to resolve a defect from the time it is open until it is closed. This metric provides insights into the efficiency of the defect resolution process and the responsiveness of the development team to reported issues.

The metric is calculated as the sum of the time each defect spent in an open state divided by the total number of resolved defects.

A lower average defect lifetime indicates a more efficient and responsive defect resolution process. Monitoring changes in this metric over time helps assess improvements or regressions in the defect resolution process.

It’s imperative that the metric is measured in a consistent time unit (e.g., days) for accurate comparisons. Different severity levels of defects may have different target resolution times, so it’s essential to consider this context too.

7. Defect Removal Efficiency

The defect removal efficiency metric is used to evaluate the effectiveness of the testing process at identifying and removing defects before the software is released. It measures the percentage of defects that were found and fixed during the development and testing phases, compared to the total number of defects present in the software. It provides insights into the quality of the testing process and its ability to identify and address defects early in the software development life cycle.

Defect Removal Efficiency = Number of Defects Found Pre-release / Total Number of Defects Found

A high percentage indicates that a significant portion of defects has been identified and addressed during development and testing, reducing the likelihood of defects reaching users in the production environment. A lower percentage suggests that a higher proportion of defects is being identified by end users after the software has been released, indicating potential gaps in the testing process.

It’s worth mentioning the factors that affect defect removal efficiency: comprehensive test coverage and effective test design, code reviews and inspections, and automation, especially for repetitive and critical test scenarios.

8. Defect Leakage Ratio

The defect leakage ratio (also known as defect escape rate or defect leakage rate) is a metric that quantifies the proportion of defects that were not identified during the testing phase and subsequently reached the end users in the production environment. It is complementary to the Defect Removal Efficiency metric and measures the effectiveness of the testing process in preventing defects from “leaking” into the live environment.

Defect Leakage Ratio = Defects Found Post-Release / Total Defects Found Pre- and Post-Release

A higher defect leakage ratio indicates a higher proportion of defects reaching end users after the release.

A lower ratio suggests a more effective testing process in identifying and fixing defects before the software is deployed.

This metric helps assess the overall quality of the software and the efficiency of the testing process in preventing post-release issues. Monitoring it over multiple releases helps identify trends and areas for process improvement.

What constitutes an acceptable defect leakage ratio can vary based on industry standards, project requirements, and the criticality of the application. The severity and impact of defects that leak into the live environment should be considered in conjunction with the ratio.

9. Denied Defects

This metric refers to the percentage of reported defects that have been reviewed and rejected/denied without being addressed or fixed. It provides insights into the defect management process and the reasons why certain reported issues are not considered valid or necessary to address. The reasons why defects were denied typically include “Not Reproducible”, “By Design”, “Duplicate”, “Won’t Fix” etc.

A certain number of denied defects are common in any development process. It’s essential to distinguish between valid defects and those that are denied for legitimate reasons. Denied defects may include issues that are out of scope, not reproducible, duplicates, or those that don’t align with project requirements or priorities.

Monitoring the denied defects metric can provide insights into the clarity of requirements, effectiveness of defect reporting, and the collaboration between development and QA teams. High denial rates may warrant closer collaboration between development and QA teams to align expectations and improve communication.

10. Defect Trends

This metric is about analyzing the number and types of defects reported over different time periods. It identifies trends in defect discovery, allowing teams to proactively address emerging issues and continuously improve processes.

One of the uses of this metric is to analyze how the number and severity of defects change over time during the different phases of development, such as unit testing, integration testing, and system testing.

Conclusion

These metrics serve as indispensable tools for assessing and enhancing the quality of software development projects. They provide actionable insights into various aspects of the development and testing processes, helping teams make informed decisions and drive continuous improvement.