Regression Testing Playbook: What to Re-Test After Every Release

Every release re-test answers one question: did this change break something that used to work? ISTQB defines a regression as a degradation in the quality of a component or system due to a change, and the trap is that the broken code is usually nowhere near the diff. You fix a rounding bug in billing and the export-to-CSV button stops working three modules over.

The lazy answer is to run all 4,000 test cases on every commit. That does not scale, it does not pay back, and at volume it actively misleads you. This playbook gives you the decision instead of the list: what to re-test, what to skip, what to automate, and what to probe by hand — grounded in the numbers that make those calls non-negotiable.

ISTQB draws a hard line here, and conflating the two is how teams ship broken fixes. Confirmation testing re-runs the exact test cases that failed last time, to verify the corrective action worked. Regression testing then checks that the same change did not degrade previously-working behavior elsewhere. Confirmation is narrow and points at the bug you just fixed. Regression is broad and points at everything around it.

Skip confirmation and you ship a fix that never actually fixed anything. Skip regression and you repair one path while silently breaking three. A complete release re-test always runs both, in that order: confirm the fix, then regress the blast radius.

A smoke test is broad and shallow. It touches the critical paths across the whole app to answer exactly one thing: is this build stable enough to bother testing further? It runs in minutes and it gates the pipeline. If smoke is red, you stop — no point running a three-hour suite against a build where login is down.

A full regression test is the inverse: narrow-but-deep across many features, verifying existing behavior still holds. It can take hours. The two are complementary, not interchangeable. Smoke catches a broken deploy before you waste a morning; regression catches the subtle behavior drift a smoke pass flies straight over.

Those two numbers are the whole argument for risk-based selection. Google's study across 5.5 million targets and 4 billion outcomes found that after filtering flaky targets, only about 1.23% of test executions actually caught a real breakage — most of a full run re-confirms things that were never at risk. Worse, roughly 84% of pass-to-fail transitions were flaky rather than genuine, so a red full suite is more often noise than signal. Run that on every commit and your team learns to ignore red, which is the exact opposite of what a test suite is for.

This is the rule that falls directly out of the data. Mature, high-frequency, rarely-changing functionality is the perfect automation candidate: it pays back its maintenance cost on every run, and it is exactly the regression you never want to re-check by hand again. Let automation guard the rear.

The code that changed in this release is where exploratory, manual testing earns its keep, because that is where behavior is genuinely uncertain. A human notices the off-by-one, the weird empty state, the error toast that fires twice — things a scripted assertion written before the change never anticipated. Automating a brand-new, still-shifting feature just produces brittle tests you rewrite next sprint. Probe the frontier with humans; automate the settled ground behind them.

DORA's 2024 report estimates that every 25% increase in AI adoption is associated with a roughly 7.2% decrease in delivery stability and a 1.5% decrease in throughput — and it pins the mechanism on larger batch sizes, not the AI-generated code itself. Bigger changes, more blast radius. GitClear's 2025 analysis across 211 million changed lines reports rising code churn (lines reverted within two weeks) and more copy-pasted code, both leading indicators that more regressions are shipping. The response is not less AI; it is smaller batches plus disciplined, impact-selected regression on every release.

A regression has something most bugs do not: a known-good baseline. That makes before-and-after the single most useful artifact for finding the change that introduced it. BugMojo's browser extension records rrweb session replay, console logs, network requests, and screenshots at the moment something breaks — so you capture the broken flow as state, not as a one-line stack trace. Put the working replay next to the broken one and bisecting to the offending commit becomes a comparison instead of a guess.

The Model Context Protocol server then hands that captured context to AI coding agents like Claude Code and Cursor, so the agent reads the failing session, console, and network state directly and locates the breaking change instead of working from a vague repro. To be clear about the boundary: BugMojo does not run or schedule your suite — your CI still owns selection and execution. BugMojo owns the evidence.