Test Coverage and Its Limits

What coverage measures

Test coverage is a measurement of how much of your code runs while the tests are executing. A tool watches the program during the test run and records which parts were reached. The result is usually shown as a percentage, with a report that highlights the lines that never ran.

The common kinds of coverage are:

• Line (or statement) coverage: the share of lines that were executed at least once.
• Branch coverage: the share of decision outcomes taken. For an if statement, both the true and the false path count separately.
• Function coverage: the share of functions that were called at all.

Branch coverage is stricter than line coverage, because a single line can contain a decision where only one outcome was ever tried.

What coverage does not measure

Here is the central point: coverage tells you what ran, not what was checked. A test can execute code without verifying anything about the result.

Consider a small illustrative function:

def average(values):
    return sum(values) / len(values)

And this test:

def test_average():
    average([2, 4, 6])

The test calls the function, so every line is executed and line coverage is 100 percent. But the test makes no assertion, so it would pass even if the function returned the wrong answer. A meaningful version would check the result, which should be 4, since 2 + 4 + 6 is 12 and 12 divided by 3 is 4:

def test_average():
    assert average([2, 4, 6]) == 4

Even that is incomplete. The function fails with an error if the list is empty, because it divides by zero. A tool would report full coverage without that case ever being tried. This is a deliberately simple illustration, but the same thing happens in large code bases: complete coverage with important behaviour untested.

Coverage also cannot tell you about:

• Missing code. If a requirement was never implemented, there are no lines to be uncovered. Coverage only looks at code that exists.
• Missing cases. Different input values often take the same path through the code but behave differently, such as boundary values.
• Interactions. Coverage from unit tests says little about whether components work together, which is the territory of integration testing.
• Quality of the expected results. If the expected value in a test is itself wrong, coverage is unaffected.

The risk of a target

Because a percentage is easy to report, it tends to become a target, and a target tends to change behaviour. If a team is told to reach a particular number, someone will write tests that raise the number cheaply, such as tests with no assertions or tests for trivial code. The figure goes up while confidence stays the same. People sometimes summarise this with the saying that when a measure becomes a target, it stops being a good measure.

A particular number is also not universally right. Some code, such as simple data structures, is not worth heavy testing, and some code, such as payment logic, deserves far more than line coverage can express. No single figure is correct for every project, and you should be wary of anyone claiming one is.

Using coverage sensibly

Coverage is most useful as a way to find gaps, not as a grade.

Look at what is uncovered. The report highlights code that no test touches. Ask whether it matters. Untested error handling for an important failure deserves attention; an unreachable defensive branch may not.

Check new code. Many teams look at coverage on each change rather than the total, asking whether the new or modified code has tests. This is often more practical than trying to raise an old project’s figure.

Prefer branch coverage when available. It reveals untested decision paths that line coverage hides.

Pair it with review. A reviewer can see whether tests have meaningful assertions, which no coverage tool will report. This is one thing to look for in code review.

Watch the trend, not the digit. A sudden fall after a change may mean tests were skipped. Small movements are usually just noise.

Don’t skip the awkward cases. Think about boundaries, empty input, errors and unexpected values, and write tests for them whatever the report says.

Other ways to judge test strength

Coverage is one signal among several. Others include:

• Whether bugs found in production later get a test so they do not return.
• Whether the tests fail when you deliberately break the code. A tool category known as mutation testing automates a version of this by making small changes to the code and checking that tests notice, though it is slower and not needed for every project.
• Whether the tests are readable and fail for clear reasons.
• Whether the checks trace back to what the software is meant to do. Linking requirements to tests shows gaps that coverage cannot, because it starts from the requirement rather than from the code.

Where it fits

Coverage reports are commonly produced by the automated pipeline in continuous integration, so the team sees them on every change. That visibility is useful as long as the number is treated as a conversation starter and not a verdict.

In regulated or safety-related work, standards may define specific coverage measures that must be met and documented, and the required kind and level differ by industry and country. In that setting, follow the standards and the qualified people responsible for them. Outside it, coverage is best seen as a flashlight: it shows where you have not looked, and says nothing on whether what you saw was right.