Test Strategy For Teams That Ship Daily

Most organisations do not have a test strategy. They have a test suite, which is a different thing: an accumulation of decisions taken by different people under different pressures over several years, with no governing idea and no mechanism for removing anything. The suite grows, gets slower, gets less trusted, and eventually acquires a coverage target imposed by someone who wanted a number to manage.

A strategy answers a question the suite cannot: what are we buying, and what are we buying it for? Every test is a purchase. It costs time to write, time to run on every commit for the rest of the system's life, and maintenance whenever the code it touches changes. In return it buys a specific kind of confidence about a specific kind of failure. A good portfolio is one where each instrument buys confidence the others cannot, and where nothing is being paid for twice.

This matters far more once you deploy daily, because the suite stops being a safety net and becomes the gate. It is the precondition on which trunk-based development rests. When a change reaches production within the hour, the tests are the only thing standing between a mistake and a customer. Teams that ship monthly can compensate for a weak suite with a hardening period and a room full of people. Teams that ship daily cannot, and should not try.

Design a portfolio, not a coverage number

Code coverage measures which lines were executed while the tests ran. It does not measure whether anything was asserted, whether the assertions were meaningful, or whether the behaviours a user depends on are protected. It is a useful signal in one direction only: very low coverage in a critical module tells you something real. A high number tells you almost nothing, and a mandated high number tells you the team has learned to write tests that execute code without examining it.

The better framing is risk. For each part of the system, ask what failure would actually cost, how likely it is, and how quickly you would notice without a test. Payment calculation and authorisation logic deserve exhaustive, fast, cheap tests. A rarely used administrative screen probably does not deserve an end-to-end suite, and the effort would be better spent on monitoring that tells you when someone uses it and it breaks.

The question that matters most is where to buy each piece of confidence, because the same defect can usually be caught at several layers and the cost differs by orders of magnitude.

The pyramid and what its critics get right

The test pyramid — many fast unit tests, fewer integration tests, very few end-to-end tests — remains the best default shape, for a reason that is purely economic. Tests at the bottom are fast, deterministic and pinpoint the fault. Tests at the top are slow, environment-dependent, and tell you only that something somewhere is wrong.

The critics are right about two things, and both matter in practice.

The first is that a heavily unit-tested system can be comprehensively wrong. Unit tests verify that each component does what its author believed it should. They cannot detect that two components hold incompatible beliefs, and mock-heavy unit tests actively conceal it by asserting against an imagined collaborator rather than a real one. A system can have superb unit coverage and fail on every real request.

The second is that the pyramid was described when integration tests were genuinely expensive. Containerised dependencies, in-memory databases and fast service virtualisation have moved the cost curve; the middle is much cheaper than it was, and many teams are correct to sit heavier there than the classic diagram suggests.

What the critics do not overturn is the ordering principle: push each check to the cheapest layer that can honestly answer the question. The pyramid was always an argument about cost of feedback, not architectural purity.

Layer	The question it answers	What it cannot tell you	Failure mode when overused
Unit	Does this piece of logic behave correctly across its input space	Whether the pieces agree with each other	Tests couple to implementation and block refactoring
Integration	Do these components work against real collaborators, schemas and queries	Whether the whole user journey holds together	Slow suites, shared fixture coupling
Contract	Do two independently deployed services still agree on their interface	Whether either side's internal logic is right	Contracts drift from reality if not generated from real tests
End-to-end	Can a user complete the critical journeys	Anything specific about where a failure originated	Flakiness, long feedback, maintenance overwhelms value
Production	Does it work for real users, on real data, at real volume	Anything before the change is live	Learning about defects from customers rather than pipelines

The practical guidance is uncomfortable but consistent: keep a very small number of end-to-end tests covering the journeys that would embarrass you if they broke — sign in, purchase, the core workflow — and resist every request to add more. End-to-end tests are the layer people ask for after an incident, and the layer that quietly degrades the suite's trustworthiness over the following year.

Flaky tests are a trust problem

A flaky test is one that passes and fails on identical code. Teams treat flakiness as an annoyance. It is not. It is the mechanism by which a test suite stops functioning as a gate, and once it takes hold the damage is not proportional to the number of flaky tests.

The sequence is always the same. A test fails intermittently. Someone reruns the build and it goes green. Rerunning becomes normal. Then a red build no longer means "something is broken"; it means "try again". At that point the suite has lost the only property that made it valuable, which was that red was information. A real defect will now be reran past by a tired engineer at five in the afternoon, and it will reach production.

Treat flakiness as a defect class with its own policy, enforced without negotiation.

Quarantine immediately. A test identified as flaky comes out of the blocking suite the same day. It keeps running, and its results are visible, but it no longer has the power to stop a build. This is not tolerance; it is preserving the meaning of red for everything else.

Give it an owner and a deadline. Quarantine without a clock is deletion with extra steps. Two weeks is a reasonable default; if nobody fixes it in that window, the honest action is to delete it and record what confidence you have given up.

Fix the cause, not the symptom. Almost all flakiness comes from a small set of causes: time and timezone assumptions, test ordering and shared state, real network calls, fixed waits instead of polling on a condition, and genuine concurrency bugs in the system under test. The last of those is the important one. A flaky test is sometimes a correct test of a racy system, and deleting it deletes a warning you were given for free.

Track the rate as a metric. The proportion of builds that fail for reasons unrelated to the change is a quality measure for your delivery system, and it belongs on the same dashboard as pipeline duration.

Contract testing at service boundaries

Once a system is split across independently deployed services, the classic strategy breaks down. Integration tests that spin up every service are slow, brittle and require a shared environment; without them, each service is tested alone and nobody verifies that the pieces still agree.

Contract testing resolves this. The consumer expresses what it needs from the provider as an executable expectation. That expectation becomes an artefact the provider verifies against in its own pipeline. The provider then learns, at build time and in its own repository, that a proposed change would break a real consumer — without either team needing a shared environment or a coordinated release.

Two points determine whether it works in practice.

The contract must be generated from real consumer tests rather than written as a hopeful specification, otherwise it documents what somebody once intended rather than what the code actually requires. And the verification must be a blocking step in the provider's pipeline, because a contract that reports failures into a channel nobody reads is documentation with a build step attached.

The organisational effect is larger than the technical one. Contract testing converts a coordination problem — which teams must test together, in what environment, before which release — into a build-time check inside each team's own boundary. That is often the specific thing standing between an organisation and independent deployment, and it removes a class of cross-team dependency that no amount of planning ceremony ever solves.

Testing in production is part of the strategy

There is a residual category of failure that no pre-production test can catch, because it depends on real data volumes, real traffic patterns, real third parties and real user behaviour. Pretending otherwise leads teams to build ever more elaborate staging environments that are expensive, permanently out of date, and still not representative.

The mature position is to design deliberately for the production half of the strategy.

Progressive delivery. Release to a small slice of traffic first — a canary deployment, a percentage rollout behind a flag, an internal cohort. Watch the error and latency signals for that slice against the rest. Automate the rollback on a threshold rather than relying on someone noticing.

Synthetic monitoring. Run your critical end-to-end journeys continuously against production as scripted transactions. This is the honest home for a great deal of what teams try to force into a pre-production end-to-end suite, and it has the advantage of testing the thing customers actually use.

Observability rather than monitoring. Monitoring tells you that a known thing has gone wrong. Observability is the ability to ask a question you had not anticipated about a running system without shipping new code to answer it. For a daily-deploying team it is not a luxury, because the failures that reach production are precisely the ones nobody predicted.

Error budgets as the governing loop. Explicit reliability targets, with the consequence that breaching them shifts work from features to stability. This is what stops "test in production" from becoming a euphemism for not testing.

None of this substitutes for a pipeline you trust. It complements it. The question to keep asking is not "could this have been caught earlier?" but "what is the cheapest place this could have been caught, and do we have an instrument there?"

What happens to the manual QA team

This is the part of the conversation that usually goes badly, because it is handled as a tooling change when it is a change to people's jobs.

The work that disappears is regression checking: re-executing a documented script to confirm that behaviour which worked last month still works. That work is repetitive, poorly suited to humans, and the reason release cycles have hardening periods. Automating it is not a cost saving to be pocketed; it is capacity released.

The work that grows is considerably more valuable and considerably harder to hire for. Exploratory testing — skilled, unscripted investigation aimed at finding the failures nobody thought to specify — has no automated equivalent, and testers who are good at it find defects that no suite will. Test design and risk analysis, deciding what deserves protection and at which layer, is a genuine specialism that most engineering teams do badly. Quality coaching, helping engineers write better tests rather than writing tests for them, scales further than any individual contribution. And someone has to own the health of the pipeline itself: flake rate, duration, and whether the suite still means anything.

The transition fails predictably: the team is told it will be retrained, given no time to do so while still owning the existing regression cycle, and gradually reduced. Handle it explicitly. Name the destination roles, fund the time, and move accountability for quality onto the delivery team from the first day rather than the last — a separate quality gate at the end is itself a queue, and a daily-deploying organisation cannot afford one. The definition of done has to change at the same time; if done still means "passed to test", nothing structural has moved.

What to do on Monday

Measure two numbers. First, pipeline duration from commit to a deployable verdict — if it exceeds about ten minutes, you have a behavioural problem, because people batch their commits around slow feedback. Second, the proportion of builds in the last month that failed for reasons unrelated to the change. That is your flake rate, and if it is not close to zero your suite is not currently a gate.

Run the twenty-defect exercise above and let the answer set your investment, rather than adding tests at whichever layer the last incident happened to touch.

Then pick the slowest, flakiest end-to-end test in the suite and ask what question it was bought to answer. In most cases you can answer that question with a fast integration test plus a synthetic monitor against production, for a fraction of the running cost and none of the flakiness. Delete it, put the two replacements in place, and use the result as the template for the rest.