Task statement 4.1 · Prompt engineering and structured output
Design prompts with explicit criteria to improve precision and reduce false positives
The CI review posts about forty comments per pull request, mostly on comment wording and variable names. Developers now resolve them unread, and last week that included one pointing at a real SQL injection.
share
The tempting fixes ask the model to be careful: "be conservative", "only report high-confidence findings", or a confidence cut-off. None of them says which kind of finding is unwanted, so the model drops real issues it is unsure of along with noise it is sure of. Meanwhile the noisy category keeps posting, and developers keep ignoring every comment, the accurate ones included.
KnowledgeWhat decides whether a finding is reported
Claude first finds candidate issues in the diff, then decides whether to report each one, and the prompt is the part of that decision you control. "Check that comments are accurate" leaves it to the model's reading of "accurate", so a comment saying "gets" where the code "fetches" looks as reportable as one promising sorted output the code never sorts. "Flag comments only when claimed behavior contradicts actual code behavior" decides the whole class, so it reports the sorting comment and skips the nit on every run. That is why precision, the share of reported findings that are real, comes from naming categories: forty comments with ten real issues is 25% precision.
- The pipeline sends the diff and the review prompt.
- Claude finds candidates: bugs, security flaws, style, local patterns and comment wording.
- A candidate whose category a criterion names is reported or skipped the same way on every run.
- Any other candidate rests on Claude's reading of words like "accurate", which can differ between runs.
Interactive · same diff, three review prompts
Choose the review prompt, and see which of the eight candidate findings reach the pull request.
All eight reach the pull request, and four are false positives: 50% precision. “Accurate” and “important” leave every candidate to the model's reading.
Three reach the pull request: 67% precision in this run. The list is shorter, but two real issues the model was unsure of were dropped, and a wording nit it was sure of still got through.
In this run the four real issues are reported and the four others are skipped. Each candidate fell into a class the prompt had already ruled on.
KnowledgeWhy "be conservative" is not a criterion
Instructions such as "be conservative" or "only report high-confidence findings" filter on how sure the model is, not on what kind of finding it is. Claude can be sure that "gets" and "fetches" differ, yet unsure about a real bug that depends on how callers use a function. So the confident nit stays, the uncertain bug goes, and the shorter list still has to be checked comment by comment. Compared with categorical criteria, that is no gain in precision. When one fix asks for caution and another names what to report and skip, choose the one that names categories.
More precise than the guide. Anthropic's prompting guides for the current Sonnet and Opus models say Claude follows "be conservative" or "only report high-severity issues" faithfully: it investigates as thoroughly, then omits findings below the bar, so precision typically rises while recall falls. Their fix matches the guide: state concretely where the bar is (checked 2026-10-01). Exam answer: confidence instructions do not fix precision; explicit categorical criteria do.
SkillCriteria that say what to report and what to skip
Write criteria that name what to report, such as bugs and security, and what to skip, such as minor style and local patterns, instead of relying on confidence-based filtering. A local pattern is a convention of this codebase, such as for…of where the diff uses forEach. The diff cannot show whether it matters, the team already knows it, and a linter enforces the conventions they care about, so a comment adds work and no information. Give the reason with each skip rule, because Anthropic's prompting guidance notes that Claude generalises from an explanation.
Asks for a quality
Review this diff. Check that comments
are accurate and flag anything important.
Be conservative. Only report
high-confidence findings.Result: the decision stays with the model, so the list gets shorter but every comment still has to be checked.
Names the categories
Flag a comment only when the behaviour
it claims contradicts what the code
actually does.
Report: bugs, security issues.
Skip: minor style and naming (the linter
enforces them) and patterns local to
this codebase.Result: each candidate falls into a class the prompt has already ruled on, so wording nits are out by definition.
SkillDisable the noisy category, then fix it
Suppose comment accuracy produced 31 of a pull request's 40 comments, and developers dismissed 28 of them. Developers judge the review as a whole, so they start resolving everything unread, including the finding about req.params.id concatenated into SQL. That is how a high false positive category undermines developer trust in the accurate ones. First, temporarily disable it by moving comment accuracy to the skip list, so the next pull request shows only bugs and security findings and developer trust can recover. Then rewrite its criterion offline, test it on the last 50 pull requests, and switch it back on when its precision matches the others. Claude Code's managed Code Review supports both moves through skip rules in REVIEW.md.
SkillSeverity defined by code, not adjectives
Severity labels fail the same way. With "blocking" defined as "serious issues", the pagination slice that drops each page's last row was blocking on Monday's run and nit on Tuesday's, so severity could not decide what stops a merge. Give each level one concrete code example, and tell Claude to classify by the closest one. The slice then matches the warning example, an off-by-one that skips the last row, so classification is consistent across runs.
## Severity levels (classify by the closest example) blocking Exploitable or data-corrupting on a normal code path. e.g. db.query("SELECT * FROM orders WHERE id=" + req.params.id) warning Wrong result on an edge case a caller can reach. e.g. for (let i = 0; i < rows.length - 1; i++) // skips the last row nit Correct but harder to maintain. Report at most five. e.g. a helper that duplicates formatDate() in utils/
- A confidence instruction filters by certainty, so real issues the model is unsure of go with the noise.
- One noisy category makes developers ignore the accurate ones: switch it off, fix it offline, then switch it back on.
- Severity labels agree across runs only when each level has a code example.