Task statement 7.2 · Troubleshooting and optimization

7.2 · Adjusting from Feedback

In Wrenfield Housing, the course example, the operations team uses Claude to draft the letters that ask tenants to let the gas safety engineer in. The letters read well and pass every check, yet more engineers find nobody home, so the team has to decide what to change and how much.

Why it's tested

The tempting move is to reword the prompt, because the prompt is the part you control and the drafts already pass review. A passing review shows that a draft is accurate, not that it works. The cause of a poor result can also sit outside the wording: in the materials Claude copies, in the timing of a step, or in using Claude at all. The opposite move fails too, because dropping Claude after one loud complaint throws away drafts that may be fine.

KnowledgeFeedback, results and a target set in advance

Two kinds of evidence tell you whether an approach is working. Feedback is what people say about the work: the readers, the colleagues who deal with what follows, and the reviewers. Results are what actually happened, measured: calls received, visits rebooked, errors found in review, or time taken. Feedback explains why something falls short, while results show whether it does and by how much. Before you change anything, decide what result will count as better: one measure, its baseline (its value before the change), a target and a date. The checks in 1.3 judged one draft; this target judges what the work achieves. Anthropic's guidance asks for success criteria that are specific and measurable. Its account of enterprise adoption describes teams that track concrete measures and refine from them. A target set in advance rules out judging a change by feel.

The adjustment then runs in five steps:

  1. Set the target: a measure, its baseline, a target and a date.
  2. Gather both kinds of evidence: ask the readers and the people who deal with what follows, and count the result.
  3. Read for patterns: how many made the same point, and do the results agree?
  4. Choose the adjustment: the smallest change that reaches the cause.
  5. Re-measure: one change at a time, against the same target (5.4).

Interactive · which size of change each piece of evidence points to

On the left, the evidence about the gas safety letters, from top to bottom in the order it arrived: the officer's check, the results, the feedback, and the re-measured results. On the right, four sizes of adjustment from smallest to largest: the prompt or instructions, the materials or examples, the workflow step, feature or model, and stopping the use of Claude. The results point to the prompt change that was tried and did not move the rate. The feedback points to the materials and to the workflow step, which were chosen. Nothing points to stopping. Evidence not yet gathered is dashed and labelled not gathered yet. EVIDENCE, IN THE ORDER IT ARRIVED SIZE OF CHANGE, SMALLEST FIRST Officer's check every letter passes: facts right, tone polite SHOWS ACCURACY, NOT WHETHER TENANTS ARE IN Result: visits with nobody home 16% before Claude → 21% with its letters after “under 150 words”: still 21% NOT GATHERED YET Feedback: tenants' panel, call handlers panel, 9 of 12: date buried, sounds optional 70 calls a month: letter came 3 days ahead NOT GATHERED YET Re-measured against the 16% target month 1, new model letter: 17% month 2, 10 working days' notice: 13% NOT GATHERED YET 1 · Prompt or instructions “under 150 words” TRIED: STILL 21% NO EVIDENCE YET 2 · Materials or examples new model letter: date first CHOSEN: 21% → 17% INDICATED BY FEEDBACK NO EVIDENCE YET 3 · Workflow, feature or model send 10 working days ahead CHOSEN: 17% → 13% INDICATED BY FEEDBACK NO EVIDENCE YET 4 · Stop using Claude here the letters are accurate NOT INDICATED NO EVIDENCE YET TRIED POINTS TO POINTS TO Each pattern points at a size of change; the target judges whether it worked.

Add the gas safety evidence in the order it arrived, and see where each piece points.

Review only: every letter passes the officer's check, so nothing here says the letters fail.

Results added: no-access visits rose from 16% to 21%. A prompt line asking for shorter letters left the rate at 21%, so length was not the cause.

Feedback added: the panel found the date buried, and call handlers heard that letters came too late. One pattern points at the model letter, the other at when letters are sent.

One change a month, against the 16% target: the new model letter gave 17%, and earlier sending gave 13%. Nothing pointed at stopping.

Dashed and labelled NOT GATHERED YET or NO EVIDENCE YET: not known at this step. Sizes run from smallest (1) to largest (4).

One loud comment is a single view, however strongly put. A pattern is the same point made by several people or confirmed by a result. A pattern earns a change, and a single comment earns a note. Anthropic's Claude 101 course says the same: note patterns and refine your prompt accordingly. A result that does not move after a change is evidence too, because it shows the change missed the cause. Another change of the same kind will probably miss it again. Some feedback points past Claude, at the process around it or at its sources, such as a letter sent too late or an out-of-date file. No prompt can fix those.

KnowledgeFour sizes of adjustment

The smallest adjustment is a change to the prompt, and the largest is to stop using Claude for that part of the work. Between them sit a change to the materials Claude works from and a change to the workflow, the steps around Claude. A bigger change costs more to make and to check, so start with the smallest one that reaches the cause. Never repeat one that the results have already ruled out. Claude 101 names both ends: add examples that show Claude what good looks like, or learn what you should not delegate at all.

Prompt or instructions

Change one line, as in 1.3 and 5.3.

Fits whenReaders name one thing missing, and the materials are sound.

Materials or examples

Replace the example or file Claude works from, as in 4.3.

Fits whenThe drafts copy a fault from what Claude is given.

Workflow step, feature or model

Move a step around Claude, or change the feature (3.1) or model (3.3).

Fits whenThe fault is timing, a handover or capacity, not the words.

Stop using Claude for that part

Give the step back to people (the fit test, 6.1).

Fits whenResults stay poor after sound changes.

SkillWorked example: the gas safety letters

Wrenfield's operations team sends about 450 letters a month asking tenants to let the gas safety engineer in for the yearly check. Claude drafts each letter in the Repairs Project from the job reference, estate, date and slot only, as AI policy rule 1 requires. An operations officer checks it and adds the name and address before it goes out (rule 2). Before Claude, 16% of visits found nobody home and had to be rebooked. With Claude's letters the rate was 21%, although every letter passed the officer's check:

# before: a prompt tweak, judged by feel
Evidence: rebooked visits 16% → 21%; nobody asked why
Change:   "Keep each letter under 150 words."
→ shorter letters, all pass the check; next month still 21%
# after: a target first, then both kinds of evidence
Target:   16% or lower within two months; letter calls halved
Panel:    9 of 12: "the date is buried", "it sounds optional"
Calls:    70 a month; letters arrive 3 working days ahead
Change 1: new model letter, date and slot first → 17%, 40 calls
Change 2: letters sent 10 working days ahead → 13%, 25 calls
→ target met; Claude still drafts, the officer still checks

The length tweak failed because it aimed at a cause nobody had found, and the feedback found two. The model letter in the Repairs Project gave the date and slot only in its third paragraph, so Claude's drafts did the same, and tenants read the visit as optional. The fault was in the materials, so the team rewrote the model letter with the tenants' panel. The call handlers' log pointed at a workflow step: letters went out when the contractor confirmed its schedule, three working days ahead, too late to swap the slot. The planning lead moved that confirmation earlier, which no prompt could do. One panel member wanted "Dear Sir/Madam" letters, a single view against the style guide, so the team noted it. The two changes went in a month apart, so each figure shows what one change did.

Ask one question for each pattern in the feedback and results: does it point at what Claude is given, or at the work around Claude? At what Claude is given → change the prompt or the materials, smallest first. At the work around Claude → change the workflow step, the feature or model, or stop using Claude for that part. Either way, re-measure against the target you set.
Takeaway
  • Feedback says why the work falls short; results say whether it does and by how much.
  • A single comment earns a note and a pattern earns a change; a result that did not move means the change missed the cause.
  • A passing review shows accuracy, not that the approach works, so judge each change against a target set beforehand.

check yourself

0 / 3 answered

A charity's fundraising team uses Claude to draft thank-you emails to donors, and the team lead checks each one. One major donor sends a long complaint that the emails feel machine-made. In the same quarter, 14 of 15 donors surveyed rate the emails as good, and repeat giving rises from 31% to 34%.Which response is most proportionate?

Select one.

A clinic's reception team uses Claude to draft the instructions patients get before a blood test, and a nurse checks each one. About one patient in five still arrives without fasting. The team added "put the fasting rule in bold" to the prompt, and a month later the rate is unchanged. Nobody has asked the receptionists or patients why.What is the most effective next step?

Select one.

A council's licensing team uses Claude in a Project to draft replies to licence applicants, and an officer checks each one. Four in ten replies lead to a follow-up email. Officers report that applicants keep asking for the application form link, which is missing from the Project's files, and that replies go out about two weeks late because the shared inbox is read only on Fridays.Which TWO adjustments best match what the evidence shows?

Select two.

sources

  1. Claude Platform Docs, "Define success criteria and build evaluations": good success criteria are specific and measurable, such as a named measure instead of "good performance". Checked 2026-10-15.
  2. Claude Academy, "Getting better results" (Claude 101): compare Claude's results to your originals; note patterns and refine your prompt accordingly; adjust your prompts and add examples that show what good looks like; learn what you should not delegate at all. Checked 2026-10-15.
  3. Claude Blog, "How enterprises are driving AI transformation with Claude" (2025-10-01): tracking concrete measures such as productivity, time savings and quality improvements, for continuous refinement. Checked 2026-10-15.
  4. Claude Help Center, "What are projects?": files in a Project's knowledge and its instructions give Claude the background for chats in that Project; Projects on all plans, sharing on Team and Enterprise. Checked 2026-10-15.
  5. Wrenfield's gas safety letters, their no-access rates, the tenants' panel and call handler feedback, and the two changes are this course's example.