QA Metrics That Matter: Escape Rate, Density, and MTTR

Open almost any QA dashboard and you'll find a wall of numbers that are easy to collect and hard to act on: test cases authored this sprint, bugs filed, tickets closed, automation runs. They feel like progress because they go up. But none of them answer the only question leadership actually cares about — is the software getting safer to ship?

The problem is a category error. Those numbers measure activity, and activity is a proxy for effort, not for quality. A team can write a thousand test cases that all exercise the same happy path, file a mountain of duplicate bugs, and still ship a release that falls over in production. To know whether QA is working, you have to measure outcomes: how many defects reach users, how fast you recover when one does, and how often a "fixed" bug comes back. This playbook covers the handful of metrics that do that, each with a plain definition, the formula, and an honest view of what "good" looks like.

If you only track one QA metric, track this one. Defect escape rate directly answers the question the whole discipline exists to answer: are we catching bugs before customers do? Every other activity metric is a means to this end.

To measure it, count the defects your team caught before release and the defects that slipped through to production over the same window — say, a release or a month. Add them for the denominator, then divide the production count by that total and multiply by 100. If you caught 90 bugs pre-prod and 10 escaped, your escape rate is 10 ÷ 100 = 10%.

The benchmark is unusually clear. Mature teams keep escape rate below 5% — the overwhelming majority of defects are caught before shipping. Teams without a deliberate QA process routinely sit at 30-50%, meaning a third to half of everything they build is discovered by users in production. The single number tells you, unambiguously, which camp you're in. And because it's a ratio rather than a raw count, it stays honest as your team and codebase grow.

Defect density, in the ISTQB definition, is the number of defects per unit size of a work product — commonly defects per thousand lines of code (KLOC), or per feature, per component, or per story point. Formula: defects ÷ size. Ten defects across a 5-KLOC module gives a density of 2 defects per KLOC.

Its value is comparison. Density normalizes for size, so you can rank modules by risk ("the billing service has triple the density of everything else — that's where regression effort should go") or track whether a rewrite actually reduced defects relative to its size. It's a targeting tool: it tells you where the risk concentrates.

The trap is using it to compare people or teams. Density is exquisitely sensitive to how you count both defects and size, and the moment it becomes a scorecard, people optimize the count — logging fewer defects, padding line counts, arguing over what's a "real" bug. Use it to point testing effort at the riskiest code. Never use it to rank engineers.

Mean time to restore (MTTR), sometimes "time to restore service," is one of the four DORA metrics and the key stability signal: when something breaks in production, how long until service is restored? Escape rate measures prevention; MTTR measures recovery, and you need both, because no team catches everything. DORA's research puts elite performers at restoring service in under an hour. This deserves its own treatment — see the companion playbook how to reduce MTTR for the deep dive on measurement, incident flow, and the tooling that moves the number. The short version: MTTR falls fastest when the person debugging doesn't have to reconstruct what happened from scratch.

Reopen rate is the share of resolved bugs that get reopened, and its close cousin is the "cannot reproduce" rate — the share of reports a developer closes because they can't recreate the problem. Formula: (reopened bugs ÷ total resolved bugs) × 100.

The instinct is to read a rising reopen rate as sloppy developers shipping half-fixes. Occasionally that's true. Far more often, it's a reporting problem wearing a developer's face. When a bug arrives without reliable reproduction context, the developer fixes what they think the bug is, the tester verifies against what they remember, and the mismatch surfaces as a reopen a week later. A high "cannot reproduce" rate is the same failure one step earlier — the developer never had enough to recreate the state at all.

That makes reopen rate a leading indicator of context quality, not diligence. If it's climbing, don't start with a conversation about developer care. Start by asking what each bug actually carries when it lands: are there exact steps, the environment, the console error, the network activity? Usually the answer is no, and the fix is upstream of engineering entirely.

Test coverage — the percentage of code (lines, branches, functions) executed by your test suite — is worth watching, but it's the metric most prone to misreading. Coverage measures what your tests execute, not what they verify. A test can run a function, assert nothing meaningful, and still light up the coverage report green.

That gap makes coverage trivially gameable: a suite of assertion-free tests can post 90% coverage while catching nothing. Treat it as a floor and a trend, not a target. Falling coverage on a critical module is a real warning. A team chasing a coverage number for its own sake is optimizing the proxy, not the quality. Read coverage alongside escape rate — high coverage and a high escape rate together is the tell that the tests are executing code without actually checking it.

Some numbers survive on dashboards purely because they're easy to collect. They should come off:

• Raw bug count. The most seductive and the most ambiguous. More bugs filed can mean testing got sharper or that code quality collapsed — and the number alone can't tell you which. A rising count with a falling escape rate is great news; a rising count with a rising escape rate is a fire. The count only means something as a ratio.
• Number of test cases. Quantity is not coverage and it's certainly not value. Ten precise tests aimed at real risk beat a thousand that re-check the same happy path. Counting cases rewards writing more of them, not better ones.
• Lines of test code. A pure verbosity metric. It rewards the least maintainable tests and tells you nothing about what they catch.

None of these predict whether a release is safe. Keep them out of the view leadership reads, where they crowd out the metrics that do.

Look closely at the metrics that matter and a common cause emerges. Reopen rate climbs when bugs lack reproduction context. "Cannot reproduce" is that same gap, named. MTTR stretches when whoever's debugging has to reconstruct what happened before they can even start. The quiet variable driving all three is the same: how much reproduction context each bug carries when it arrives.

That's the metric worth engineering. When a bug lands with a session replay, the console log, and the network waterfall already attached, the developer doesn't re-derive the state — they watch it. The fix targets the real defect, so it holds, so the reopen rate falls. Recovery starts at the evidence instead of an hour of reconstruction, so MTTR drops. The escape rate improves too, because a tester who can capture a flaky issue completely files it instead of shrugging it off as unreproducible.

This is the whole idea behind BugMojo: every captured bug arrives complete — an rrweb DOM replay, console, network, and environment — so nobody has to remember or re-derive what happened. You don't move reopen rate and MTTR by exhorting people to be more careful. You move them by making sure the context was recorded, not recalled.