Task statement 2.4 · Output evaluation and validation

2.4 · When Human Review Is Required

In Wrenfield Housing, the course example, Claude drafts everything from internal brainstorms to figures for the regulator. The AI policy says a person checks anything that goes to tenants, the board or the regulator. It does not say how much checking each draft needs, so one draft nearly reached the regulator with only its writer's read-through.

Why it's tested

The tempting habit is to give every draft the same check. A quick read for everything lets a wrong figure reach a regulator, because the writer reads for wording and cannot see the source system. Sign-off for everything turns reviewers into a queue, so they skim, and the drafts that matter get skimmed too. The other trap is trusting how sure the draft sounds. The Claude Help Center warns that Claude can write things that look correct but are very mistaken, so a confident tone tells you nothing about whether a check is needed.

KnowledgeWhat decides how much checking an output needs

Human review is a person reading an output before it is used or sent. Additional verification goes further: someone checks the output's claims against a source outside the conversation, such as the system a figure came from. Both cost time, so the amount should follow what is at stake. Anthropic's AI Fluency course says that whoever uses or shares an output takes responsibility for verifying and vouching for it, and that different contexts carry different expectations. It gives no fixed scale, so the questions below are common practice, not an Anthropic rule. They rule out any answer that chooses the check by the output's format, such as "it is only an email", or by how confident the draft sounds.

  1. Who receives it? Only you, your own team, or people outside it: customers, tenants, the board, a regulator or the public.
  2. What happens if it is wrong, and can it be undone? A team note can be corrected next week; a posted letter or a filed return cannot be called back.
  3. Is it regulated, legal, financial, or about a specific person? These outputs carry duties beyond being correct, and some need a qualified reviewer under Anthropic's own policy.
  4. Does it hold facts Claude could not know? Your organisation's figures that were never in the conversation, or events after the model's knowledge cutoff.
  5. Must an expert sign it off? Some outputs need a person with the knowledge and the authority to answer for every claim.

Interactive · four outputs, four levels of checking

On the left, a draft from Claude passes down through three checks before it is used or sent: a self-check, a second person's review, and expert verification. On the right, the questions about what is at stake, with the chosen output's answers. A check the chosen output does not need is dashed and labelled not needed; a check it needs is solid and labelled required. THE DRAFT'S PATH Claude's draft written from your prompt and files PASSES TO 1 · Self-check the writer, against notes and criteria NOT NEEDED REQUIRED PASSES TO 2 · Second person's review a colleague reads it as the reader will NOT NEEDED REQUIRED PASSES TO 3 · Expert verification checked against the source, then signed NOT NEEDED REQUIRED RELEASED Used or sent by you, your team, or a reader outside WHAT IS AT STAKE WHO RECEIVES IT? choose an output above only the writer the operations team 1,200 Millbrook households the regulator IF WRONG, CAN IT BE UNDONE? nothing acts on it yes: corrected at next week's meeting no: posted letters stay posted no: it goes on the record REGULATED, LEGAL, FINANCIAL, PERSONAL? no no, but it reaches every tenant yes: a regulatory return FACTS CLAUDE COULD NOT KNOW? none relied on only figures from the export dates and slots, all from the brief figures from the repairs system EXPERT SIGN-OFF? no yes: the operations director SETS LEVEL OF CHECKING set by the answers above none: use it as it is 1: self-check 1 and 2: the communications lead reads it 1, 2 and 3: second check and director Each level adds to the one above it; none replaces it.

Choose one of Wrenfield's outputs and see which checks it needs before it is used or sent.

Brainstorm: only the writer reads it and nothing acts on it, so it needs no review. A bad idea is simply not chosen.

Team update: the operations team acts on it, but can correct it next week, so the writer's self-check against the export is enough.

Tenant letter: it reaches 1,200 households and cannot be called back, so a second person, the communications lead, reads it first.

Regulator return: it is regulated, goes on the record and holds figures Claude could not check, so a second check against the repairs system and the director's sign-off come first.

Dashed and labelled NOT NEEDED: a check this output can skip. Solid and labelled REQUIRED: a check it must pass.

The fourth question needs the most care, because no self-check can answer it. Claude knows only what the conversation and its training give it, and each model has a knowledge cutoff after which it may not know about events. When a draft states a figure from your own systems, or a recent fact, rereading it, by you or by Claude, checks the wording, not the fact. Only the source can confirm the claim, and how to check against a source is the subject of 2.3.

KnowledgeThree kinds of check

A self-check is the writer reading the draft against their notes and criteria. Anthropic's prompting guide adds a cheap step: ask Claude to verify its answer against stated criteria before it finishes. A self-check catches slips, but it shares the writer's blind spots and Claude's. A second person's review is a colleague who did not write the draft reading it as its reader will, so they see gaps the writer no longer can. Expert verification is a person with the training and authority in the field checking the claims against the original source and signing the result. Each kind adds to the ones before it:

Self-check

The writer, against the notes and criteria, plus "verify your answer against these criteria" in the prompt.

MissesAnything the writer and Claude both got wrong, and any fact neither can see.

Second person's review

A colleague who did not write it reads it as the recipient will.

MissesA wrong figure that reads well, unless the reviewer opens the source.

Expert verification

A qualified person checks each claim against the original source and signs off.

CostsThe most time, so keep it for high-stakes outputs.

SkillWorked example: the covering note that was nearly sent

Each quarter Wrenfield sends the regulator a data return with a short covering note. In October an operations officer asked Claude to draft the note, with last quarter's note as the example and the figures attached. The officer saw the result as an email rather than part of a return, so they read it once and queued it to send:

# before: sorted by format
Output: an email to the regulator → self-check, then send
→ "All damp and mould reports are inspected within 10 working days."
# after: sorted by what is at stake
Receives it: the regulator. Undone? No, it goes on the record.
Regulated: yes. Facts Claude could not know: inspection times.
→ Level 3: second check against the repairs system; director signs
→ the system shows a median of 15 working days; the claim is cut

The cause was sorting the draft by its format. The inspection sentence was in no attachment, and the backlog export has no inspection field, so Claude wrote a plausible line in the style of last quarter's note. The officer's read-through could not catch it, because the sentence read well and no source was open. The operations director, copied in, stopped the email. Wrenfield now sorts anything sent to the regulator as level 3, whatever its format.

Anthropic's rule, not only practice. Anthropic's Usage Policy, the rules for using its products, lists high-risk areas such as housing, legal, finance and employment. When Claude's output is a recommendation about an individual in those areas, a qualified person must review it before it is delivered or acted on. A new version of the policy takes effect on 12 November 2026, and both versions keep this requirement (checked 2026-10-15).

Ask one question: if this output were wrong, who would it reach, and could you take it back?
Only you, and nothing acts on itNo review: use it as it is.
Your team, and it can be correctedSelf-check against your notes and criteria.
People outside the team, and it cannot be called backAdd a second person's review before it goes.
Regulated, legal, financial or about a person, or built on facts Claude could not knowAdd expert verification against the source, with sign-off.
Takeaway
  • Choose the check by the stakes, not by the format of the output or how confident Claude sounds.
  • Rereading a draft, by you or by Claude, checks the wording; only the source can confirm a figure Claude could not see.
  • Self-check, second person's review and expert verification add to each other; a higher level never skips a lower one.

check yourself

0 / 3 answered

An operations analyst asks Claude to summarise last week's warehouse incident log, which they pasted into the chat, for their own team's Monday meeting. The team will discuss each point and can correct anything that looks wrong. Nothing in the summary leaves the team.Which is the most proportionate check before the analyst shares the summary?

Select one.

A payroll officer uses Claude to draft a letter telling one employee that a £1,240 overpayment will be recovered from their next three payslips. The amount comes from the payroll system, which Claude cannot see, and the letter cites a clause in their employment contract as the basis for recovery.Which TWO checks does this letter most need before it is sent?

Select two.

A shop's social media posts are drafted with Claude and checked only by the person who wrote them, the same way as internal notes. The brief pasted into Claude said a sale ends on 30 June. The post said 3 July, and customers who bought on 2 July were given the discount.What is the most likely root cause of the error reaching customers?

Select one.

sources

  1. Claude Help Center, "Claude is providing incorrect or misleading responses. What's going on?": Claude can write things that might look correct but are very mistaken; do not rely on Claude as a singular source of truth; scrutinise high-stakes advice; review cited sources. Checked 2026-10-15.
  2. Claude Help Center, "How up-to-date is Claude's training data?": each model has a knowledge cutoff and may not be aware of events after it. Checked 2026-10-15.
  3. Claude Academy, "A closer look at Diligence": Deployment Diligence means taking responsibility for verifying and vouching for the outputs we use or share; different contexts have different expectations for verification. Checked 2026-10-15.
  4. Claude Platform Docs, "Prompting best practices": ask Claude to self-check, for example "Before you finish, verify your answer against [test criteria]". Checked 2026-10-15.
  5. Anthropic, "Usage Policy": high-risk areas including housing, legal, finance and employment; a qualified person must meaningfully review a high-risk recommendation before it is delivered or used for a decision. The version effective 12 November 2026 and the previous version (effective 15 September 2025) both require this. Checked 2026-10-15.
  6. The questions about stakes and the three kinds of check are common review practice, not an Anthropic rule. Wrenfield's levels of checking, the covering note and its inspection figures are this course's example.