Putting AI In The Delivery Pipeline

Most organisations have deployed coding assistants in exactly one place: an engineer's editor, doing completion. That is the obvious application and it is not the highest-value one. It attacks the production half of the transaction cost, which, as the transaction cost argument sets out, is the half that was already falling and is no longer the constraint.

The interesting applications are the ones that attack the verification half, or that remove fixed costs from the release path. Those are the ones that reduce lead time rather than increase inventory. They are also, predictably, the ones that get adopted last, because they require somebody to change a process rather than install a plugin.

There is a selection rule that sorts the good applications from the bad, and it is worth having before the list. An application pays back when the output is cheap to verify or when a wrong answer is cheap. It is a trap when the output is expensive to verify and a wrong answer is expensive — because in that case you have moved effort from a step you could see to a step you cannot, and the verification cost lands on your constraint. Apply that rule to every proposal and most of them sort themselves.

The map, sorted by payback

Application	Verification cost	Cost of being wrong	Verdict
Large mechanical migrations	Low — compiler and existing tests	Low — caught immediately	Pays back immediately
Triage and routing	Low — a human sees it next anyway	Low — misrouted, then rerouted	Pays back immediately
Review assistance as a pre-pass	Low — advisory only	Low — a human still reviews	Pays back immediately
Drafting decision records from real discussion	Low — participants recognise it	Low — corrected in place	Pays back immediately
Test generation from specification	Moderate — assertions must be read	Moderate	Pays back if disciplined
Explaining unfamiliar code	Moderate — plausible and wrong is possible	Moderate	Useful with scepticism
Test generation from implementation	High	High — false confidence	Trap
Generating documentation nobody reads	Low	Low, but the value is also nil	Waste, not a trap
Automated approval or gate-passing	Very high	Very high	Trap
Incident diagnosis taken as authoritative	High	Very high	Trap

The pattern is consistent. Wherever the machine's output lands in front of a human who was going to look anyway, it is close to free money. Wherever the output substitutes for a human judgement that was providing assurance, you have removed the assurance and kept the artefact.

The ones that pay back immediately

Large mechanical migrations. Framework upgrades, API deprecations, logging or configuration standardisation, consistent renames across hundreds of files. This is the strongest case available and it is systematically underused because it is not a daily activity. The reason it works is that verification is nearly free — the code either compiles and passes the existing suite or it does not, and the changes are individually trivial to read. Migrations that were never economic before, because they required six weeks of tedium nobody would fund, are now economic. Many of them are sitting in your technical debt register, priced at an effort level that is no longer accurate. That register is worth repricing.

Triage and routing. Summarising an incoming defect, identifying probable owning service, linking likely duplicates, extracting reproduction steps from a rambling report. The human still decides; the machine removes the reading. This is queue-reduction work on a queue that is nearly always unmanaged, and being wrong costs a reroute.

Review assistance as a pre-pass. A machine pass over a change before a human opens it, flagging candidate issues. It catches a class of routine problems and it costs your constraint nothing. The discipline is that it is advisory — it is not the review, it does not know your invariants, and it cannot hold accountability.

Decision capture from real discussions. Turning an actual architectural argument into a structured record — options considered, what was decided, what was rejected and why. The participants can recognise whether it is accurate in about thirty seconds, so verification is cheap, and the artefact is genuinely valuable later. This is the one application where volume genuinely helps: decision records were always a good idea and were never written because writing them was tedious. That obstacle has gone.

The traps

Tests generated from the implementation. The dominant trap, and the most dangerous because it improves every metric you are likely to watch. Point a generator at existing code and ask for tests, and you get a suite that asserts the behaviour the code currently has, including the behaviour that is wrong. Coverage rises. Confidence rises. Assurance does not move at all, because the suite is a mirror rather than a check. What you have built is a very effective regression detector for a specification you never stated, and a reason for reviewers to stop reading carefully.

Anything that passes a gate automatically. If an assistant summarises a change for a change advisory board, and the board approves on the summary, you have replaced a human reading a change with a human reading a description of a change. The gate still exists, still costs a fortnight, and now provides less assurance than it did. Either the gate does real work, in which case do not automate its input, or it does not, in which case delete it rather than making it cheaper to satisfy.

Incident diagnosis treated as authoritative. During an incident, a confident and wrong hypothesis is worse than no hypothesis, because it directs scarce attention down a specific path at the moment attention is most expensive. Use it to summarise timelines and surface related changes; do not let it propose a root cause to a room that is under pressure and looking for one.

Documentation generated at volume. Not dangerous, merely worthless. Documentation that describes what the code does is redundant with the code. Documentation that has value describes why, what was considered and rejected, and what constraints the reader cannot see. That information is not in the source and cannot be derived from it.

Test generation, done in the order that works

Test generation is worth separating out because it is simultaneously the highest-value and the highest-risk application, and the difference is entirely about sequence.

Specify first, in prose. State what should be true: invariants, boundaries, error behaviour, the properties that hold for all valid inputs. This is the step people skip and it is the step that carries all the value.

Generate tests from that specification, with the implementation out of scope. The tests must be derivable from the stated behaviour alone. If a test can only be written by looking at the implementation, it is testing implementation.

Read every assertion. This is not optional and it is the actual cost of the technique. Reading assertions is far cheaper than writing them, which is where the saving comes from, but it is not zero and it must be budgeted.

Prefer properties to examples where you can. Property-based testing states an invariant and lets a generator hunt for counterexamples. It is a better fit than example-based testing here for a structural reason: properties are derived from the specification by construction, so the failure mode of mirroring the implementation is much harder to fall into.

Confirm the suite can fail. Deliberately break the behaviour and check that something goes red. A suite that has never been observed failing is not evidence of anything, and this takes ten minutes.

Where the system under test is not deterministic, none of the above applies directly and you need the equivalent machinery for probabilistic output — evaluation harnesses and the associated practice of testing non-deterministic systems.

Provenance, logging and what a regulated estate has to show

If you operate under any kind of external scrutiny — financial services, health, safety-critical, public sector procurement — you will be asked about this, and the question will be asked badly. Expect something like "do you use AI in your development process, and how do you control it?" A good answer requires having thought about it in advance.

The framing that holds up is the one already established in agile in regulated environments: regulators require evidence of control, not a particular control. The relevant controls here are mostly ones you should already have.

Accountability is unambiguous and documented. A named human submits every change and is accountable for it regardless of how it was produced. Write this down as policy. It is the single most important sentence in your response and it costs nothing, because it is already true.

Review is evidenced and human. Your existing records already show who approved what and when. The addition is a written statement that approval requires a human judgement and that machine assistance is advisory.

Test provenance is traceable. You can show which requirement a test derives from. This matters more than usual, because the mirror-testing failure mode is exactly the thing an auditor should worry about, and being able to demonstrate specification-first derivation answers it directly.

Data handling is explicit. What source code, customer data or configuration leaves your boundary, to which processor, under what contractual terms, with what retention. This is a procurement and data protection question rather than an engineering one, and it should be answered by the same people who answer it for every other processor.

Dependency and licence provenance is unchanged. Whatever you already do to establish that third-party code is licensed appropriately continues to apply. Generated code that resembles something with a restrictive licence is a genuine question and your existing scanning is the existing answer.

What to do on Monday

Take your technical debt register and reprice it. Find every item whose cost was dominated by mechanical tedium across many files, and re-estimate it honestly. Some of those items have become weekend work. Pick the one with the best ratio of value to remaining risk and do it, as a demonstration that the payback is real and mechanical rather than aspirational.

Separately, audit where assistants are currently used in your lifecycle and sort each use with the rule at the top of this article: how expensive is it to verify the output, and how expensive is it to be wrong. Anything in the high-high quadrant should stop this week. Anything that automates a process you should have deleted should prompt a conversation about deleting it instead.

Then write the accountability sentence down and publish it. One line in your engineering standards saying that the submitting engineer owns every line of every change they submit, however it was produced. You will need it for the audit conversation, and you will need it more for the post-incident one.