DORA Metrics Without The Gaming

The four key metrics from the DORA research programme — deployment frequency, lead time for changes, change failure rate and failed deployment recovery time — are the best-evidenced measures of software delivery performance in general use. They come out of a multi-year research effort, popularised through Accelerate, which found that these four discriminate between higher and lower performing organisations and correlate with organisational outcomes rather than merely with engineering activity.

They are also, in my experience, the metrics most reliably destroyed by the organisations that adopt them. Usually within two quarters, and usually by exactly the people who were most enthusiastic about adopting them.

The destruction follows a predictable sequence. Someone reads the book. A dashboard is commissioned and rolled out per-team. Someone notices the teams have different numbers. The numbers appear in a quarterly review sorted descending. Targets are set. Within a quarter the numbers have improved and delivery has not, because the measurement system has become a compensation system and everyone has responded rationally.

This article is about how to get the value without that ending. It requires three things: precise definitions, an understanding that the four only function as a set, and a reporting discipline that makes league tables structurally impossible.

Why they work as a set and collapse individually

The four metrics are usually described as two pairs: deployment frequency and lead time measure throughput; change failure rate and recovery time measure stability. That framing is correct but it understates the point.

The reason the set works is that each metric is trivially improvable in isolation, and every isolated improvement degrades at least one of the others.

• Maximise deployment frequency alone and you get trivial deployments padding the count, or a drop in pre-deployment verification.
• Minimise lead time alone and you get smaller units of measured work and skipped review.
• Minimise change failure rate alone and you get fewer deployments, longer testing phases and larger batches — the exact opposite of what you want.
• Minimise recovery time alone and you get incidents closed prematurely and reclassified as something other than incidents.

Each of these movements is visible in the other three. That mutual constraint is the entire mechanism. The four metrics are not four measurements; they are one measurement with four projections, and it is only the shape of all four together that carries information.

The practical consequence is a rule with no exceptions: never report one of the four without the other three, and never set a target on one. A team asked to improve deployment frequency will improve deployment frequency. A team asked to improve all four simultaneously has been asked to genuinely improve, because there is no cheap way to move all four at once — the only reliable route is smaller batch size, better automation and better test strategy, which is the actual point.

Getting the definitions right

Most measurement failures are definition failures, and they happen quietly. Here are the definitions that matter and the specific places instrumentation goes wrong.

Deployment frequency. How often you deploy to production. Not to staging, not a merge to main, not a release candidate. The unit that counts is a change reaching real users.

Where it goes wrong: counting pipeline runs rather than production deployments; counting a multi-service deployment as one event when it is twelve independent ones, or as twelve when it is a single coupled release; lumping automated dependency bumps in with functional change. Also, very commonly, averaging a bimodal distribution — one service deployed forty times and eleven deployed never averages to something meaningless.

Lead time for changes. Time from code committed to code running in production. This is the narrow engineering definition and it is deliberately narrow. It is not idea-to-production, which is a different and also valuable measure — that one belongs in your flow metrics as flow time.

Where it goes wrong: measuring from merge to main rather than from first commit, which hides all the time a change spent on a branch — often the majority of it. Measuring from ticket creation, which inflates the number with backlog age and makes it useless for diagnosing the delivery path. Reporting a mean rather than a distribution, which hides the tail where all the pain is.

Change failure rate. The proportion of deployments to production that result in degraded service requiring remediation — a rollback, a hotfix, a patch, a forward fix under urgency.

Where it goes wrong: this is the one that requires the most discipline, because the numerator is a judgement call made by the people being measured. Organisations dispute what counts as a failure, exclude "config issues" or "environment issues", and quietly drift toward only counting severity-one incidents. The denominator matters too: it is deployments, not changes and not time. A rate of ten percent means nothing without knowing whether that is ten failures in a hundred deploys or one in ten.

Failed deployment recovery time. How long from a failed deployment to restored service. Note that the current DORA framing is specifically about recovering from deployment-caused failures, which is narrower and more useful than a general mean-time-to-restore across all incidents including hardware and third-party outages.

Where it goes wrong: starting the clock at detection rather than at the failure, which rewards poor monitoring by making slow detection invisible. Stopping the clock at "mitigated" when the user impact persisted. Reporting a mean over a small number of incidents, where one bad night dominates everything.

Across all four, the same instrumentation principle applies: derive from systems of record — version control, deployment pipeline, incident tooling — not from anything a human types into a form. Self-reported metrics measure reporting behaviour.

Goodhart's Law, metric by metric

Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. It is not a warning about dishonesty. It is a structural observation about what happens when you place an optimising system in front of a proxy. People do not need to be cynical for this to happen; they need only be responsive.

Here is the specific pathology each metric invites, and the counter-signal that exposes it.

Metric	How it gets gamed	Counter-signal that exposes it
Deployment frequency	Empty or trivial deployments; splitting one release into many; counting non-production environments	Lead time flat or rising while frequency climbs; deployments with no code diff
Lead time for changes	Measuring from merge not first commit; shrinking the unit of "a change"; bypassing review	Branch age; review depth; change failure rate rising
Change failure rate	Reclassifying incidents; deploying less often; longer manual test phases; raising the severity bar	Deployment frequency falling; recovery time rising; incident volume flat while failure rate falls
Recovery time	Closing incidents early; starting the clock late; declaring "mitigated" as "recovered"	Repeat incidents on the same service; customer-reported issues outnumbering monitored ones

Two structural defences beyond the counter-signals.

Make the denominators and definitions owned centrally, and the numbers computed automatically. If a team can influence the definition, the definition will move. Publish the query, not just the result. Anyone should be able to see exactly how a number was derived and dispute the method rather than the figure.

Report change failure rate and recovery time as a lagging pair against frequency and lead time. Plot them on one chart, over time, on the same team. Gaming any single metric produces a visible divergence in the shape. A genuine improvement produces all four moving together, slowly, which looks much less impressive on a quarterly slide and is worth infinitely more.

What the four do not measure

Worth saying plainly, because the most dangerous failure is not gaming but over-scope. The four key metrics measure the health of your delivery pipeline. They say nothing whatsoever about:

Whether you built the right thing. A team can achieve elite delivery performance shipping features nobody wants, quickly and reliably. That is a product problem and requires outcome measures.

Whether the people are alright. Delivery throughput can be raised temporarily by burning people. The SPACE framework exists partly for this reason: satisfaction and wellbeing are a dimension of performance in their own right, not a nice-to-have alongside it. If your four metrics improve while attrition and on-call load rise, you are consuming capital, not generating it.

Whether the architecture is sustainable. Recovery time can be excellent in a system held together by a small number of heroic individuals who know where everything is. That is a risk register item, not a capability.

Whether the organisation can decide. In many enterprises the dominant delay is not in the pipeline at all — it is upstream, in funding, prioritisation and approval. A pipeline measured from first commit cannot see any of it. If lead time is excellent and things still arrive late, stop optimising the pipeline and go and measure the part before the commit.

Reporting so that nobody can build a league table

This is a design problem, and it is solvable. The goal is not to hide the numbers — hidden numbers get reconstructed badly from worse sources. The goal is to present them in a form that makes ranking obviously invalid to the person reading.

Aggregate at the level you intend to act. If the decision you will take is an investment in platform or tooling, report at the organisation or value-stream level. Per-team numbers are for the team, in the team's own retrospective, for the team's own use. This is not secrecy; it is matching the granularity of the report to the granularity of the decision. If an executive asks for per-team numbers, ask what decision they will make with them. Frequently there isn't one, and the request evaporates when tested.

Always show all four, always as trends, never as a snapshot. A single-period number invites comparison. A twelve-month trend line invites the question "what changed in March?", which is the question you want.

Use distributions, not averages, for the time-based metrics. Show the median and the eighty-fifth percentile for lead time and recovery time. This is honest, it is more useful for forecasting, and as a side effect it makes tidy ranking harder because there are two numbers moving.

Report context alongside. System age, domain constraints, regulatory scope, whether the team owns its own deployment path. A payments team constrained by a certification cycle should have that stated on the same page as its numbers, every time, permanently.

Pair delivery metrics with at least one people measure and one outcome measure. Even a crude one. The presence of a wellbeing signal and a customer signal on the same page changes how the delivery numbers are read, because it makes it obvious that the delivery numbers are a means rather than an end.

Give the narrative to the team. Each cycle, the team writes two or three sentences on what moved and why. This is the part executives actually use, and the part no pipeline configuration change can game. It also converts the review from an interrogation into a conversation, which is the difference between a metrics programme that survives and one quietly abandoned after a year of people learning to hate it.

What to do on Monday

Instrument before you discuss. Pull deployment events from your pipeline and commit timestamps from version control for the last six months, for one value stream, and compute lead time as a distribution. Do not build a dashboard yet. Build one chart and look at it.

Write the four definitions down, with the exact query for each, and get them agreed by engineering leadership before any number is shown to anyone outside. Include explicitly what does not count — the exclusions are where all the future arguments live, and settling them in advance while nothing is at stake costs you an hour and saves you a quarter.

Find your change failure rate honestly by going through the last thirty deployments individually. It takes an afternoon. The automated version will disagree with you, and the discrepancy will tell you more about your incident classification than the metric will.

Then present the first report with all four, as trends, at value-stream level, with the team's own narrative attached and no targets. If someone asks for a target, offer a direction of travel and a review date instead. The distinction sounds like semantics and is not: a target is something to be met, and a direction is something to be pursued.