Task statement 5.5 · Context management and reliability
Design human review workflows and confidence calibration
The pipeline extracts 10,000 documents a week, and reviewers check its output. The dashboard says 97% of extractions are correct, and the proposal is to stop reviewing the ones the model is confident about.
share
The tempting move is to trust the two numbers the pipeline already reports. The overall 97% is a mean dominated by the largest document type, so a small type that fails badly disappears into it. The confidence score looks like a probability, but until it is compared with labelled answers nobody knows how often a 0.9 is right. Cutting review on either number removes the check on the extractions that are wrong.
Start hereThree terms this section leans on
- segment
- Extractions measured on their own: one document type, or one field within it, such as
requested_amountin support emails. - labelled validation set
- Documents with a human-recorded correct value for every field.
- calibration
- Setting a field's review threshold from that set, so each confidence score maps to an observed accuracy.
KnowledgeWhat an overall accuracy figure hides
Accuracy here is the share of labelled documents whose extracted fields all match the human answer. Measured over everything, it is a mean weighted by volume, so each document type counts in proportion to its size. Here 7,000 invoices at 99.0% and 2,000 contracts at 97.5% outvote 1,000 support emails at 85.0%, giving 97.3%. At 85%, about one support email in seven is extracted wrongly, and those emails carry the refund amounts customers ask for. So accuracy is validated by document type and field segment, and must be consistent across all segments, before any high-confidence extraction is automated.
- Measure accuracy by document type, then by field within any failing type.
- Have the model output a confidence score for each field.
- Calibrate each field's threshold on a labelled validation set.
- Route low-confidence fields, and ambiguous or contradictory documents, to human review.
- Automate only passing segments, and keep stratified random sampling of them.
Interactive · the same 10,000 extracted documents, measured three ways
Choose how accuracy is measured, and see whether review can safely be reduced.
One number, 97.3%, clears the bar. Nothing here says which document types it is made of.
Split by document type, about one support email in seven is extracted wrongly. The 7,000 invoices outvoted them in the overall figure.
Within support emails, one field fails. Reviewer time can go to requested_amount instead of to every support email.
SkillCalibrating field-level confidence, then routing
The field view shows that requested_amount causes most support-email errors, so review can target that field instead of every support email. Choosing which extractions of it to trust needs a field-level confidence score, and the raw score is only the model's own estimate. Calibration turns it into a measurement: run the extractor over 500 labelled support emails, group the extractions by score, and count how often each group was right.
score band extractions actually correct
0.95 – 1.00 280 97%
0.85 – 0.95 140 86%
below 0.85 80 61%
Scores between 0.85 and 0.95 are right only 86% of the time, below the 95% bar, so the threshold goes at 0.95. A requested_amount scored under it goes to a reviewer. So does a document whose source is ambiguous or contradictory, such as an email naming both $40 and $65 as the refund, because any single value the model picks is a guess. Reviewer capacity is limited, so when the queue outgrows it, contradictory sources and the lowest-confidence fields go first, because that is where errors are likeliest.
SkillSampling what nobody reviews any more
Automating a segment does not end measurement, because a confident extraction can still be wrong and no reviewer sees it. Stratified random sampling keeps measuring its error rate: it splits the high-confidence extractions into segments, document type by field, and draws at random within each one. Each segment usually gets a fixed number of reviews (disproportionate allocation), because a sample proportional to volume gives small segments the same handful a simple random sample does. Of 210 weekly reviews, that handful is about 147 invoices, 42 contracts and 21 support emails, against a fixed 70 of each.
The fixed allocation is what detects novel error patterns. In week 12 a large retailer starts forwarding support emails with an older order thread quoted underneath, and the extractor takes the old order number with confidence 0.97. About 100 of the week's 1,000 support emails are affected, and confidence routing never sees them. The two samples see that week differently:
210 reviews drawn from all 10,000
About 21 support emails, 2 with a wrong order_number. That looks like noise, and the overall figure falls about one point, still above 95%.
70 reviews within each document type
70 support emails, about 7 with a wrong order_number, in a segment that usually shows none. The new pattern is found in its first week.
Not the same as self-reported confidence. Section 5.2 rejects the model's self-reported confidence as a trigger for escalating a conversation. Here a score routes review only after calibration against labelled answers turns it into a measured accuracy. An uncalibrated score is as unreliable here as there.
- 97% overall can contain a document type at 85%, because the large types outvote the small one.
- A confidence score routes nothing until a labelled validation set has shown what each score band is actually worth.
- Confidently wrong extractions never reach the review queue by themselves, so sample a fixed number from every segment.