Forecasting When Throughput Changes

Every empirical forecasting method in delivery rests on an assumption that is almost never stated: that the recent past and the near future are drawn from the same distribution. Throughput forecasting, Monte Carlo simulation over historical cycle times, velocity-based projection, reference class forecasting — all of them are inference from samples, and all of them are valid exactly as long as the system generating those samples is stable.

That assumption has become questionable in a specific way. Not because delivery has become unpredictable in general — most of the sources of variability in a delivery system are structural and have not moved — but because one term in the system is changing on a timescale comparable to the forecasting window. Production capability is shifting quarter on quarter, tool adoption is uneven across teams and across types of work, and the parts of the work that got faster are not the parts your historical data separates out.

The honest position is uncomfortable and worth stating plainly: forecasts built on data from a period when the system behaved differently are less reliable than they were, and the standard response of quietly continuing to publish them with the same confidence interval is a misrepresentation. The right response is not to abandon empirical forecasting. It is to shorten the window, widen the intervals, be explicit about the instability, and change what you measure so that the parts of the system that did not move are separated from the parts that did.

Why historical throughput stops predicting

Statistically, this is a non-stationarity problem. Your sample is drawn from a distribution whose parameters are moving, so the sample mean is an estimate of something that no longer exists.

Three things make it worse than a straightforward level shift.

The change is uneven across work types. Work that is dominated by mechanical implementation got substantially faster. Work dominated by decisions, coordination, domain ambiguity or cross-team dependency did not. If your historical throughput is a single number across a mixed backlog, that number is now a weighted average of two populations whose weights vary with whatever happens to be in your backlog this quarter. Forecast error rises even if you re-baseline the average correctly.

The change may not reach the customer. As what AI does not change sets out, faster production does not produce faster delivery when the constraint is downstream. So you can see a genuine change in one measure and no change at all in the measure that matters, and a naive re-baseline on the wrong one will produce forecasts that are confidently wrong in the optimistic direction.

Adoption is a moving target. Unlike a process change that lands on a date, capability here is drifting continuously — new tooling, changing practice, teams at different points on a learning curve, and an initial period where things get slower before they get faster. There is no clean before and after to baseline against.

Re-baselining, without pretending

The instinct is to re-baseline: throw away old data, use only recent data, publish a new number. That is half right and the other half matters.

Shorten the window, deliberately. If you were forecasting on twelve weeks of history, move to six or eight. You lose statistical power and you gain relevance, and under non-stationarity that is the right trade. Be explicit that you made it.

Widen the intervals rather than narrowing them. This is the counterintuitive part and the part that gets dropped under pressure. Less data and a moving system means more uncertainty, not less, even if the central estimate has improved. A forecast that says "faster, and we are less sure" is honest. A forecast that says "faster, with the same confidence" is not, and it is the one most organisations will publish.

Forecast the constraint, not the whole system. If verification is your bottleneck, the throughput that matters is verified-and-released throughput. Measure the service rate of the constraint and forecast on that. It is the number that governs delivery and, conveniently, it is usually the number that has been most stable.

Keep the old baseline visible. Do not delete history. Keep both series on the chart with the re-baselining point annotated. You will need it to answer whether the change was real, and you will need it more when someone asks in six months whether the investment paid back.

Re-baseline on a schedule, not on a good week. Set a cadence — quarterly is usually right — and re-baseline then, whatever the numbers say. Ad hoc re-baselining is indistinguishable from selecting your baseline to produce the answer you wanted, and after a while it becomes exactly that.

What happens to estimation practice

The argument that story points are a measure of relative effort and never a measure of performance is unaffected by any of this; velocity is not a performance metric and was not one before. What has changed is the stability of the mapping from points to elapsed time, which was always the weakest link in the practice and is now weaker.

If a team estimates in relative effort and the relationship between effort and duration is moving for some work types and not others, the historical points-to-duration conversion degrades. Teams notice this as estimates being inconsistently wrong — sometimes wildly pessimistic, sometimes not — which is worse than being uniformly wrong, because uniform error can be corrected with a multiplier and inconsistent error cannot.

Three responses, in order of how much I would recommend them.

Move to counting items. Right-size work so items are roughly uniform and small, then forecast on throughput in items per week. This was already the better practice for the reasons set out in flow metrics. It is now clearly better, because it sidesteps the effort-to-duration mapping entirely. The reference class becomes "items we have finished recently" rather than "hours we thought something would take".

Estimate the parts that did not move. If your work is dominated by coordination, waiting and decisions, estimate those and treat implementation as small. This will feel strange for about a month and then feel accurate.

Keep points, recalibrate more often, publish ranges. If your organisation is structurally committed to points, the minimum viable response is to shorten the calibration window and stop reporting single-point estimates upward. This is a holding position, not a good one.

What I would avoid entirely is asking teams to estimate as though the assistant does not exist, or as though it makes everything faster by some blanket factor. Both produce a number that encodes a belief rather than a measurement, and both will be wrong in whichever direction is most expensive.

Which measures survive and which distort

Measure	Status	Why
Lead time for change	Survives, more important	End to end, constraint-inclusive, no way to fake it with volume
Deployment frequency	Survives	Still a good proxy for batch size and pipeline health
Change failure rate	Survives, watch closely	The early warning for verification falling behind production
Time to restore service	Survives	Unaffected by generation speed
Cycle time distribution	Survives	Use percentiles; the tail is where the queue shows
Flow efficiency	Survives, more revealing	Directly shows whether gains reach the customer
Work in progress	Survives, more important	The first place a production surge becomes visible
Throughput in items	Survives if items are right-sized	Needs re-baselining; still the best forecasting input
Velocity in points	Degrades	The effort-to-duration mapping is unstable
Commits, lines, pull requests raised	Actively misleading	Measures the station that is no longer the constraint
Assistant acceptance rate	Misleading as a delivery metric	Tells you about tool use, nothing about delivery
Coverage percentage	More misleading than before	Trivially inflatable by generated tests that assert nothing

The pattern is that measures of flow survive intact and measures of activity have become actively dangerous. This was always true; the danger level has risen sharply, because activity metrics are now easy to move by a large factor without moving delivery at all. Any target set on one will be met, immediately and comprehensively, and the gaming dynamics around DORA metrics apply with more force than usual.

Two pairings are worth watching together rather than separately, because each is only interpretable in the light of the other. Deployment frequency against change failure rate tells you whether you are going faster or just shipping more defects. Throughput against work in progress tells you whether production gains are reaching the customer or accumulating as inventory — if work in progress is rising while throughput is flat, you have found the queue, and Little's Law tells you what happens to cycle time next.

Reporting upward without overclaiming or hiding

This is the genuinely hard part, because the incentives pull in both directions at once. Overclaim and you will be held to a number you cannot sustain. Underclaim and a real improvement goes unfunded, or worse, someone else's overclaim sets the comparison.

Report outcomes, not activity, and say why. Lead time, deployment frequency, change failure rate, flow efficiency. When asked for output numbers, explain in one sentence that output measures the station that is no longer the constraint, and that the organisation's capacity to deliver is set downstream of it.

Name the instability explicitly in the pack. A single line stating that the forecast window has been shortened and intervals widened because the system is changing. This costs you nothing in credibility if you say it first, and costs you a great deal if you say it after a miss.

Separate the two claims. "Production capacity has increased materially" and "delivery to customers has got faster" are different claims with different evidence. Present them separately. If the first is true and the second is not, that is the most valuable finding you have and it points directly at the investment case for fixing the constraint.

Attribute carefully. You cannot separate the effect of tooling from every other change you made in the same period, and you should not pretend otherwise. If someone requires a clean attribution, the honest answer is that a controlled experiment would require a comparable team doing comparable work without the tooling, and that you have not run one. Resist the invitation to produce a percentage. A fabricated attribution is the fastest way to lose the argument permanently when it is later examined.

Show the leading indicator before the lagging one. Work in progress and review queue length move weeks before lead time does. Putting them in the pack lets you report a developing problem while it is still a forecast rather than a result, which is the only circumstance in which reporting a problem is comfortable.

What to do on Monday

Split your last two quarters of completed work into two buckets: items dominated by implementation, and items dominated by coordination, decisions or dependencies. Compare the cycle time distributions. If the first bucket has moved and the second has not, you have your answer about where the gain landed and where the constraint now sits, and you have it from your own data.

Then shorten your forecasting window, widen your intervals, and annotate the re-baselining point on the chart rather than quietly restating the series. Put the old and new baselines side by side in the next report.

Finally, go through your current metrics pack and strike out every activity measure. Replace each with the nearest flow equivalent. Do this before somebody sets a target on one of them, because the moment a target exists on an activity measure, the measure stops being data and becomes an instruction — and it is an instruction to increase the arrival rate at a constraint you have not yet fixed.