Task statement 4.1 · Prompt engineering and structured output

Design prompts with explicit criteria to improve precision and reduce false positives

The CI review posts about forty comments per pull request, mostly on comment wording and variable names. Developers now resolve them unread, and last week that included one pointing at a real SQL injection.

Scenario switchThis section runs on the guide's Claude Code for Continuous Integration scenario. Claude Code reviews each pull request in CI, and the pipeline posts its findings as inline comments. In extraction, a value the document does not support is a false positive too.
Why it's tested

The tempting fixes ask the model to be careful: "be conservative", "only report high-confidence findings", or a confidence cut-off. None of them says which kind of finding is unwanted, so the model drops real issues it is unsure of along with noise it is sure of. Meanwhile the noisy category keeps posting, and developers keep ignoring every comment, the accurate ones included.

KnowledgeWhat decides whether a finding is reported

Claude first finds candidate issues in the diff, then decides whether to report each one, and the prompt is the part of that decision you control. "Check that comments are accurate" leaves it to the model's reading of "accurate", so a comment saying "gets" where the code "fetches" looks as reportable as one promising sorted output the code never sorts. "Flag comments only when claimed behavior contradicts actual code behavior" decides the whole class, so it reports the sorting comment and skips the nit on every run. That is why precision, the share of reported findings that are real, comes from naming categories: forty comments with ten real issues is 25% precision.

  1. The pipeline sends the diff and the review prompt.
  2. Claude finds candidates: bugs, security flaws, style, local patterns and comment wording.
  3. A candidate whose category a criterion names is reported or skipped the same way on every run.
  4. Any other candidate rests on Claude's reading of words like "accurate", which can differ between runs.

Interactive · same diff, three review prompts

Eight candidate findings from one diff, four real and four not, filtered by three different review prompts, with counters for findings reported and false positives REVIEW PROMPT SENT WITH THE DIFF — choose a prompt above — “Review this diff. Check that comments are accurate and flag anything important.” “… Be conservative. Only report high-confidence findings.” Report bugs, security, and comments that contradict the code. Skip style, naming, local patterns. 8 candidate findings in the diff SECURITY SQL built by concatenating req.params.id REAL ISSUE REPORTED COMMENT Comment says “gets”; the code “fetches” NOT AN ISSUE REPORTED skipped BUG order.total read before the null check REAL ISSUE REPORTED STYLE Variable named tmp in a 6-line helper NOT AN ISSUE REPORTED skipped COMMENT Comment claims sorted output; code never sorts REAL ISSUE REPORTED skipped LOCAL PATTERN forEach where the team usually writes for…of NOT AN ISSUE REPORTED skipped BUG Pagination slice drops the last row of each page REAL ISSUE REPORTED skipped COMMENT TODO references a ticket that is now closed NOT AN ISSUE REPORTED skipped posted as inline comments REPORTED · FINDINGS – of 8 8 of 8 3 of 8 4 of 8 FALSE POSITIVES · FINDINGS – of 4 4 of 4 1 of 4 0 of 4 Four candidates are real issues and four are not. The prompt decides which of them developers see. 8 reported · 4 false positives · precision 50% 3 reported · 1 false positive · precision 67% 4 reported · 0 false positives · precision 100%

Choose the review prompt, and see which of the eight candidate findings reach the pull request.

All eight reach the pull request, and four are false positives: 50% precision. “Accurate” and “important” leave every candidate to the model's reading.

Three reach the pull request: 67% precision in this run. The list is shorter, but two real issues the model was unsure of were dropped, and a wording nit it was sure of still got through.

In this run the four real issues are reported and the four others are skipped. Each candidate fell into a class the prompt had already ruled on.

Illustrative run. Time reads downward: prompt, eight candidates, then what gets posted. Green rows are reported real issues, red rows are reported false positives, and dimmed rows were skipped. The conservative prompt shortens the list by dropping real issues. Explicit criteria remove whole classes of noise, but Claude still judges each candidate, so a real run can misfile one.

KnowledgeWhy "be conservative" is not a criterion

Instructions such as "be conservative" or "only report high-confidence findings" filter on how sure the model is, not on what kind of finding it is. Claude can be sure that "gets" and "fetches" differ, yet unsure about a real bug that depends on how callers use a function. So the confident nit stays, the uncertain bug goes, and the shorter list still has to be checked comment by comment. Compared with categorical criteria, that is no gain in precision. When one fix asks for caution and another names what to report and skip, choose the one that names categories.

More precise than the guide. Anthropic's prompting guides for the current Sonnet and Opus models say Claude follows "be conservative" or "only report high-severity issues" faithfully: it investigates as thoroughly, then omits findings below the bar, so precision typically rises while recall falls. Their fix matches the guide: state concretely where the bar is (checked 2026-10-01). Exam answer: confidence instructions do not fix precision; explicit categorical criteria do.

SkillCriteria that say what to report and what to skip

Write criteria that name what to report, such as bugs and security, and what to skip, such as minor style and local patterns, instead of relying on confidence-based filtering. A local pattern is a convention of this codebase, such as for…of where the diff uses forEach. The diff cannot show whether it matters, the team already knows it, and a linter enforces the conventions they care about, so a comment adds work and no information. Give the reason with each skip rule, because Anthropic's prompting guidance notes that Claude generalises from an explanation.

Vague

Asks for a quality

Review this diff. Check that comments
are accurate and flag anything important.
Be conservative. Only report
high-confidence findings.

Result: the decision stays with the model, so the list gets shorter but every comment still has to be checked.

Explicit

Names the categories

Flag a comment only when the behaviour
it claims contradicts what the code
actually does.
Report: bugs, security issues.
Skip: minor style and naming (the linter
enforces them) and patterns local to
this codebase.

Result: each candidate falls into a class the prompt has already ruled on, so wording nits are out by definition.

SkillDisable the noisy category, then fix it

Suppose comment accuracy produced 31 of a pull request's 40 comments, and developers dismissed 28 of them. Developers judge the review as a whole, so they start resolving everything unread, including the finding about req.params.id concatenated into SQL. That is how a high false positive category undermines developer trust in the accurate ones. First, temporarily disable it by moving comment accuracy to the skip list, so the next pull request shows only bugs and security findings and developer trust can recover. Then rewrite its criterion offline, test it on the last 50 pull requests, and switch it back on when its precision matches the others. Claude Code's managed Code Review supports both moves through skip rules in REVIEW.md.

SkillSeverity defined by code, not adjectives

Severity labels fail the same way. With "blocking" defined as "serious issues", the pagination slice that drops each page's last row was blocking on Monday's run and nit on Tuesday's, so severity could not decide what stops a merge. Give each level one concrete code example, and tell Claude to classify by the closest one. The slice then matches the warning example, an off-by-one that skips the last row, so classification is consistent across runs.

## Severity levels (classify by the closest example)
blocking  Exploitable or data-corrupting on a normal code path.
          e.g. db.query("SELECT * FROM orders WHERE id=" + req.params.id)
warning   Wrong result on an edge case a caller can reach.
          e.g. for (let i = 0; i < rows.length - 1; i++)  // skips the last row
nit       Correct but harder to maintain. Report at most five.
          e.g. a helper that duplicates formatDate() in utils/
Before you ship a review instruction, ask: could two careful reviewers read it, look at the same diff, and disagree about what to report or how severe it is? Yes → it is a vague instruction. Name the categories to report and to skip, and give each severity level a code example. No → it is already a criterion, and adding "be conservative" will not make it more precise.
Takeaway
  • A confidence instruction filters by certainty, so real issues the model is unsure of go with the noise.
  • One noisy category makes developers ignore the accurate ones: switch it off, fix it offline, then switch it back on.
  • Severity labels agree across runs only when each level has a code example.

check yourself

0 / 3 answered

In the CI review, the performance category's findings, such as memoizing cheap functions, are dismissed 70% of the time, while null-safety findings are accepted 95% of the time. Developers now bulk-resolve every review comment. A rewritten performance criterion cannot be tested against past pull requests until next sprint, and developers must read this week's comments again.What is the most effective immediate step?

Select one.

The CI review of Terraform changes uses the prompt "Flag anything risky; label each finding blocking or non-blocking." It posts about 25 comments per pull request, mostly on local tag-naming and ordering conventions. Blocking findings fail the build, and an open security group was blocking on one run, non-blocking on the next. Budget allows one review run per pull request.Which TWO changes best make the comments precise and the build gate consistent?

Select two.

A team replaced its error-handling review prompt with "Be conservative. Only report high-confidence findings." Comments per pull request fell from 30 to 9, but developers still dismiss 4 of the 9, and an audit found two unhandled promise rejections the old prompt had reported. The fix must not lose findings like those rejections.Which change most directly raises precision while keeping findings like the missed rejections?

Select one.

sources

  1. Claude Platform Docs, "Prompting best practices", "Be clear and direct" and "Add context to improve performance": explicit instructions over vague prompts, the colleague test, and "Claude is smart enough to generalize from the explanation". Checked 2026-10-01.
  2. Claude Platform Docs, "Prompting Claude Sonnet 5", "Code review harnesses": "only report high-severity issues" or "be conservative" is followed more faithfully, precision typically rises while recall can fall, and "be concrete about where the bar is". The same advice appears in "Prompting Claude Opus 5". Checked 2026-10-01.
  3. Claude Code Docs, "Code Review", "REVIEW.md": skip rules for paths and finding categories, a nit cap, and redefining what Important means. Checked 2026-10-01.
  4. The "claimed behavior contradicts actual code behavior" criterion, "be conservative" and "only report high-confidence findings", the report and skip categories, disabling high false positive categories, and severity criteria with code examples: Claude Certified Architect — Foundations Exam Guide v1.0, Domain 4.1. Scenario: the guide's Claude Code for Continuous Integration. Finding counts and code in the examples are illustrative.