The Three Jobs Of Labelling

Two people agree to "do the labelling". One of them means a slow, careful exercise producing a few hundred examples that must never change once finished. The other means a bulk job that should be as large and as cheap as they can make it. Neither says which, because to each of them it is obvious.

They are both right. They are describing different jobs. The plan they agree is incoherent, and the incoherence will not surface for months, because nothing about it produces an error.

This is not a communication problem and it is not a seniority problem, though it is usually diagnosed as one or the other. One ordinary English word is doing three unrelated jobs, and two of them have directly opposing requirements. Separating them takes five minutes and is the prerequisite for everything else — including evaluation harnesses for LLM systems, which tells you how to build a golden set properly and assumes you have already worked out that a golden set is what you want.

One system, three jobs

A spam filter is a convenient place to see all three without changing domain.

Measuring. Somebody sits down with five hundred emails and decides by hand which are spam. They are not building anything. They are producing a yardstick: examples where the right answer is known because a person determined it, against which the filter's answers can be compared.

Teaching. Somebody assembles a large pile of emails already marked spam or not and feeds it to a system so it learns the pattern. Nobody is measuring anything. This is raw material.

Operating. The filter is running. An email arrives and the system decides, in milliseconds, and moves it. No person is involved. This is the product doing its job.

All three involve something being labelled. Two of them produce a file of labelled examples that looks identical from the outside — same columns, often the same people, often the same afternoon's work.

The requirements are opposite

	Measuring	Teaching	Operating
Purpose	Find out whether the system is right	Make the system right	Serve a customer now
Who labels	A person, slowly, once	A person or a source, in bulk	The machine, continuously
Wants to be	Small, careful, representative, frozen	Large, varied, cheap	Fast, consistent, explainable
Changes over time	Never, once frozen	Grows constantly	Every second
If it is wrong	You cannot tell whether anything works	The system learns the wrong thing	A customer gets a wrong answer today
How you find out	You do not — this is the danger	Accuracy is poor, eventually	Someone complains

Read the third row twice. Measuring wants small and frozen; teaching wants large and growing. Those are not two points on a spectrum, they are contradictory instructions. A team given one undifferentiated brief to "do the labelling" will optimise for one of them and quietly fail at the other, and which one depends on nothing more than who happens to be doing it.

The bottom row is the one that determines how much this costs you. Bad training data eventually shows up as a system that performs poorly. Bad operating labels show up as a complaint. Bad measurement data shows up as nothing at all.

The question that separates them

Is this data being used to judge the system, or to build it?

Judging is measuring. Building is teaching. Operating is rarely confused with either for long, because it is visibly a product rather than a spreadsheet.

The confusion that matters is always between judging and building, and the question resolves it in four seconds. If the person who owns a set cannot answer it immediately, that is not a small gap to correct politely later. That is the finding.

Why mixing them is worse than it sounds

The leak

Suppose the five hundred hand-labelled emails get folded into the training pile. It is an easy thing to do and it feels wasteful not to — they are the cleanest examples anyone has.

Now measure the filter against those same five hundred. It does extremely well, because it was shown the answers. The score rises. Nothing about the filter improved, and the only instrument capable of telling you that has just been destroyed.

This is why the term of art is a held-out set. Held out is literal: kept away from the people and the processes building the thing. The closest everyday parallel is an audit sample. If the people being audited choose the sample, or see it in advance and prepare, the audit still produces a number. It is simply no longer information.

A contaminated measurement does not break. It flatters.

That asymmetry is the whole reason this belongs in a library like this one rather than in a style guide. A broken thing announces itself and gets fixed. A flattering thing gets quoted in a board pack.

Contamination is also rarely a decision anybody makes. It happens when a dataset is copied into a shared folder, when a new engineer joins and uses everything available, when a vendor's model has already seen your public benchmark, or when the same person labels in the morning for measurement and in the afternoon for training and keeps one spreadsheet. The defence is structural rather than motivational: separate files, separate owners, and a stated rule that one is never an input to the other.

The drifting ruler

A gentler failure, and more common. The measurement set is not leaked, but it keeps changing. Cases get added as new situations appear, ambiguous examples get dropped, labels get corrected by someone who now disagrees with them.

Every individual change is an improvement. The aggregate effect is that this month's score cannot be compared with last month's, because they were taken against different rulers. You have lost the ability to tell whether the system is improving, which was the only reason to measure it.

Freezing is the remedy, and it has to be a specific dated act rather than a general intention. A frozen set can be replaced — sets do go stale, and a set that no longer resembles current traffic is its own problem — but it is replaced deliberately, with the old and new both run for a period so the two scales can be related to each other. Quietly editing in place gives you neither the old scale nor the new one.

The missing set

The simplest and most common of the three. A number is reported and there is no labelled set behind it at all. It came from a template, a reasonable-looking estimate, an earlier run of something else, or a demonstration that was never wired to a real system.

Nobody necessarily lied. "Ninety-four per cent" is the kind of figure that goes into a draft as a placeholder and stops looking like a placeholder after about a week.

A number with no labelled set behind it is not a measurement. It is a claim.

The habit that prevents it costs one sentence. Never report a score alone; report the set with it. Not "ninety-four per cent" but "ninety-four per cent on the five-hundred-email set, frozen in the spring." The second sentence cannot be written by accident, which is precisely its value.

The same three jobs outside software

Spam filtering can make this feel like a narrow engineering concern. Take loan applications instead.

• Measuring: a credit officer reviews two hundred past applications and records what the correct decision would have been, to test whether an automated scorer agrees.
• Teaching: ten years of historical applications and their outcomes, used to build the scorer.
• Operating: the scorer deciding on an application that arrived this morning.

Same three jobs, considerably higher stakes, and one additional reason to keep the first two apart. The historical outcomes carry whatever biases the past decisions carried. The measurement set reflects what a credit officer believes correct today. Those are different things, and keeping them in separate files is the only arrangement in which the difference between them ever becomes visible. Merge them and the system is graded by the same standard that produced it.

Reading a number you did not produce

Most people who need this article will never label anything. They will sit in a review where somebody reports a score, and the useful skill is knowing which four questions to ask.

Ask	What a weak answer sounds like
What was this measured against?	"The test data." Which test data, how many items, labelled by whom.
How many examples are in the set?	No number, or a number that moves between reports.
Was any of it used to build the thing?	"Only some of it." Then it is not a measurement.
Is the set frozen?	"We keep it up to date." Then this month and last month are not comparable.

None of these require technical knowledge, and all four should be answerable in seconds by anyone who owns the number. Hesitation on the third is the one to pay attention to.

What to do on Monday

Pick one number your organisation currently reports about a model or an automated decision. Find out what it was measured against, how many examples that set contains, who labelled them, whether any of it was used in building the system, and when it last changed.

One of three things will happen. The answers arrive immediately and the number is real, which is the best outcome and the least likely on a first attempt. The answers arrive after some digging, in which case write them down next to the number and make that the house format. Or there are no answers, in which case you have learned something considerably more valuable than the number was.

Then read evaluation harnesses for LLM systems, which covers how to build the set properly — curation, rubrics, judging, and getting it into the pipeline — and evaluating agents for the harder case where the thing under test is a sequence of decisions rather than a single answer.