Smoke Testing vs. Regression Testing
Concept
Smoke and regression testing answer different questions at different points in the deploy lifecycle, and confusing them leads teams to either run an expensive suite where a cheap one would do, or trust a cheap check to catch what only a thorough one can. Smoke testing (also called build verification testing) asks a narrow question as fast as possible right after a deploy: "is the build usable at all?" It exercises the smallest set of critical paths — can the app boot, does the health-check endpoint respond, can a user log in — and is designed to fail loudly and immediately on catastrophic breakage, before anyone invests time in deeper testing against a build that's fundamentally broken. Regression testing asks a much broader question, independent of any specific deploy event: "did this change break something that used to work?" It re-runs a comprehensive suite covering existing functionality, specifically to catch a class of bug that unit tests for the new change would never catch — an old feature silently broken as a side effect of touching shared code.
The network/infra world has its own name for the smoke-test half of this: Post-Implementation Verification (PIV), the post-deploy check with an accompanying rollback trigger if it fails — same underlying concept as a smoke test, phrased around the rollback decision rather than the test itself (see Post-Implementation Verification (PIV) for that angle).
Tradeoffs
| Dimension | Smoke test | Regression test |
|---|---|---|
| Question answered | Is the build minimally alive | Did anything that used to work break |
| Scope | Narrow — a handful of critical paths | Broad — as much of existing functionality as the suite covers |
| Speed | Seconds to low minutes | Minutes to tens of minutes, depending on suite size |
| When it runs | Immediately after every deploy, before anything else | On every code change, or on a schedule/pre-release gate, independent of deploy timing |
| Failure tolerance | Zero — a failure should block or auto-rollback the deploy immediately | Higher — failures get triaged, not necessarily an automatic hard stop |
| What a failure tells you | The deploy itself is broken (bad build, bad config, service won't start) | A specific piece of existing behavior regressed, often unrelated to what was just intentionally changed |
The core tradeoff is the same speed-versus-coverage tension that shows up throughout testing strategy: a smoke test's entire value is being fast enough to run on every single deploy without slowing releases down, which only works because it deliberately covers almost nothing. A regression suite's value is coverage breadth, which only works because it doesn't have to run inside the tight window of "did this specific deploy succeed."
When to use / when not to
- Run a smoke test after every deploy, no exceptions — its cost is low enough that skipping it to save time is never a good trade; a deploy that silently fails to boot and isn't caught for hours is far more expensive than the seconds a smoke test costs.
- Run the full regression suite on a cadence that matches the team's risk tolerance and CI budget: on every PR for a suite fast enough to afford it, or on a merge-to-main / pre-release gate for a suite too slow to run per-commit. Don't conflate "regression suite is slow" with "skip it" — slow means schedule it deliberately, not drop it.
- Keep smoke tests minimal on purpose. The temptation to fold a few "important" functional checks into the smoke suite because they seem critical erodes the thing that makes a smoke test useful — its speed. If it takes several minutes to run, it's drifted into being a small regression suite wearing a smoke test's name, and it needs a different SLA than a post-deploy gate.
- A previously-fixed bug is exactly the kind of thing that should live in the regression suite, not be re-verified ad hoc — the whole point of regression testing is that "this was already fixed once" isn't a guarantee it stays fixed as unrelated code changes around it.
Common pitfall
Letting the regression suite grow unbounded without pruning, until it's slow enough that people stop running it locally and stop trusting a red result — assuming it's "probably flaky" rather than a real regression. This is the same trust-erosion failure mode that shows up in an inverted test pyramid: once a suite's signal becomes noisy enough to be routinely ignored, its coverage stops mattering because nobody acts on its failures. The fix isn't "run it less" — it's investing in making the suite fast and reliable enough (parallelization, tagging flaky tests for quarantine and fixing them, pruning genuinely obsolete cases) that a failure is still treated as real information.
The mirror-image pitfall on the smoke-test side is skipping it, or running it manually/occasionally, because "the deploy pipeline already has other checks." A smoke test that isn't automated and gated to every deploy provides none of its value — the whole premise is catching catastrophic breakage before it reaches users, which only works if it's unconditional.
Engineering Lens
The real design decision isn't "smoke test or regression test" — it's matching each test tier's SLA to the moment in the pipeline it protects. A smoke test's job is to be the tripwire that stops a genuinely broken build from ever serving traffic, so its bar is speed and unconditional execution, not coverage. A regression suite's job is confidence that existing behavior survived a change, so its bar is coverage and reliability of signal, and it's allowed to be slower because it doesn't gate the deploy-to-live window the way a smoke test does. Treating them as one undifferentiated "post-deploy testing" bucket is what produces either a smoke test too slow to gate deploys, or a regression suite too thin to actually catch regressions — the distinction exists specifically to avoid that tradeoff being made by accident.