Developer/Languages/Python/TestingInPython/Fundamentals/property-based-testing-with-hypothesis-vs-example-based-testing

Property-Based Testing with Hypothesis vs. Example-Based Testing

Concept

Most pytest suites are example-based: pick a handful of inputs you can think of, compute the expected output by hand, and assert the function produces it. That has a structural blind spot — the test only covers cases the author imagined, and the inputs that actually cause production incidents (the empty string, negative zero, a Unicode surrogate, a list with one element, a list with a billion) are precisely the ones nobody thought to write by hand.

Hypothesis takes a different approach: instead of asserting on specific inputs, you assert a property that should hold for a whole class of inputs — "encoding then decoding returns the original value," "sorting a list never changes its length," "the function never raises for any string input." Hypothesis then generates hundreds of randomized inputs per test run (100 by default) covering that class, actively hunting for a counterexample. When it finds one that fails, it doesn't just report the random input it happened to land on — it runs a shrinking phase, repeatedly simplifying the failing input while it still reproduces the failure, until it reaches something close to minimal (e.g. reducing a failing 776,837 down to 1, or a failing 40-character string down to the 2 characters that actually matter). That minimal counterexample is what gets reported, which is what makes the failure debuggable instead of just "test failed on some big random blob."

Hypothesis integrates directly into pytest via the @given decorator plus st.* strategies (st.integers(), st.text(), st.lists(st.integers()), and composable builders for custom types) — a Hypothesis test is a normal pytest test function, just parameterized by a generator instead of a literal list of cases.

Tradeoffs

Approach	Coverage	Speed / cost	Debuggability
Example-based (plain pytest)	Only the cases the author thought to write; systematically misses inputs nobody anticipated	Fast — a handful of fixed assertions run once each	High — the failing input is exactly the literal in the test, no extra step needed
Property-based (Hypothesis)	Broad — empirically increases branch coverage over example-based tests alone by generating adversarial inputs (edge cases, boundary values, malformed shapes) the author wouldn't have written by hand	Slower — 100 generated cases per test by default vs. one; adds real wall-clock time to a suite	Requires shrinking to be useful, but once shrunk, often finds a genuinely novel bug class the author never considered

Hypothesis is not a replacement for example-based tests — the two solve different problems and are meant to be run together: keep specific, human-legible examples for the requirements you already know about (they double as documentation of intended behavior), and add a property-based test for the invariant that should hold regardless of the specific input.

When to use / when not to

Reach for a property-based test when the function under test has a genuine invariant that's easy to state but whose violating input is hard to think of by hand — round-trip serialization (encode/decode, save/load), idempotency ("calling twice is the same as calling once"), a sort/dedup function's output properties, or any parser/validator that has to handle arbitrary untrusted input.
Keep example-based tests for anything where the expected output is a specific business requirement, not a general property — "this exact input should produce this exact formatted string" is naturally example-based; forcing it into a Hypothesis property adds complexity without adding coverage.
Don't add Hypothesis to every test in a suite reflexively — the added generation and shrinking cost is real, and a function with no meaningful cross-input invariant (most trivial getters, most glue code) gets nothing from it beyond a slower suite.
Watch for flakiness from non-determinism: because Hypothesis's search is randomized, it can surface a failing case on one run and not the next, and time-based assertions (a deadline check that passed at 199ms and fails at 201ms on a different run) can produce false "flaky" reports that have nothing to do with the code under test — pin down or widen timing-sensitive assertions before wrapping them in @given.
Introduce it incrementally on new or already-fragile code (parsers, serializers, anything handling external input) rather than retrofitting an entire legacy suite at once — the value is highest exactly where inputs are least controlled.

Common pitfall

Writing a property that's either too weak to catch real bugs (asserting something trivially true for any input, like "the function doesn't crash," when the real requirement is behavioral) or too strong and accidentally re-implements the function under test inside the assertion (making the test pass whenever the implementation matches itself, tautologically, rather than checking real behavior against an independent invariant). A well-chosen property references something structurally simpler than the function itself — "decode(encode(x)) == x" checks a round trip without re-deriving what encode should output for a given x.

Engineering Lens

Property-based testing doesn't replace judgment about what to test — it changes where that judgment gets applied. Example-based testing asks "what specific inputs do I expect this to handle correctly," which is a question about known requirements. Property-based testing asks "what must always be true about this function's behavior, no matter the input," which is a question about the function's actual contract — and answering it well often surfaces gaps in the contract itself (does this parser guarantee anything about malformed input, or does "handle gracefully" mean something specific and testable?). The real cost isn't the slower test run; it's the up-front thinking required to state a property that's both true and strong enough to be worth checking. Teams that skip that thinking and bolt @given onto arbitrary existing tests get slower suites without meaningfully better bug coverage.