AI Code Review Benchmark: What to Measure

What should an AI code review benchmark measure?

An AI code review benchmark should measure serious-defect recall, blocking precision, false blockers per pull request, contradiction rate, completion rate, repeated-run stability, latency, and cost on a blinded corpus of real changes. Keep failed reviews in the denominator and bind every expected defect to evidence that an independent maintainer can adjudicate.

That scorecard answers the operational question: can this reviewer become part of a merge gate without missing important defects or exhausting maintainers with noise? A leaderboard that reports only how many comments matched a reference does not answer it. Neither does a product demo where every pull request was chosen because the reviewer already caught something impressive.

If you are still choosing a product, use theAI code review tool decision guideto narrow the field. Then benchmark the shortlist against your repositories before you make any reviewer a required check.

An AI code review benchmark is not a highlight reel

Code review outputs are unusually easy to present selectively. A reviewer can find a real defect and still be unfit for a merge gate because it misses other serious issues, reports too many unsupported blockers, fails on large changes, or produces a different verdict on the next run. The impressive comment is evidence about one case. It is not a reliability claim.

Current public benchmarks make different choices because they answer different questions. The open-sourceCode Review Benchmaintains an offline set of 50 pull requests from five open-source projects with human-verified golden comments, then uses an LLM judge to match tool findings to those comments. Its online track samples fresh pull requests and uses later human fixes as another signal. The project explicitly identifies training-data leakage as a limitation of a fixed corpus.

The 2026c-CRAB research benchmarkuses a different oracle. It converts human review concerns into executable tests, asks a coding agent to act on each reviewer's comments, and checks whether the repaired code passes. The paper reports 184 pull requests and 234 validated review comments in the final corpus. That design tests whether review feedback can lead to a change that passes a human-review-derived executable test, rather than whether two comments use similar wording.

Neither design is the one true benchmark. Golden-comment matching is easier to run across many tools. Test-based evaluation gives a more direct oracle where the concern is faithfully captured by executable behavior, though test fidelity remains a limitation. Repository pilots can add policy questions that public leaderboards cannot know: which severities may block, how failures are reported, and whether the result describes the current pull request head.

Define the merge decision before choosing metrics

Start with the decision the benchmark must support. “Which model writes the best comments?” is too vague. A maintainer needs a narrower contract such as: “Should this candidate replace the current reviewer for blocking correctness, security, and reliability defects on our backend pull requests?”

The contract fixes four variables before the first run:

1. Scope. Name the repositories, languages, change sizes, and risk classes the conclusion may cover.
2. Blocking policy. Decide which finding classes and severities may affect merge readiness. Keep style and optional maintainability advice outside the blocker denominator.
3. Comparison. Freeze exactly one baseline and one candidate for a replacement test. Change one review policy or pipeline at a time.
4. Rollout thresholds. Set minimum recall and precision, a maximum false-blocker budget, completion requirements, and resource limits before seeing the candidate result.

Precommitting the thresholds prevents a common failure: moving the goalposts after an expensive candidate produces an attractive but mixed report. The result can still teach you something when it fails. It cannot quietly redefine success.

Build a blinded corpus that represents your pull requests

A benchmark corpus should preserve the information available at review time. For a historical defect, capture the vulnerable diff, bounded surrounding files, applicable repository instructions, and relevant tests as they existed before the fix. Keep the later fix and adjudication outside the reviewer's context.

Blinding matters. If the expected finding, fix commit, issue title, or regression test appears in the prompt, the benchmark measures answer recovery rather than code review. Reviewers should receive the same pull request subject they would see in production, not the answer key.

Include both expected defects and verified non-findings. A corpus with only broken pull requests can measure recall, but it cannot tell you how often a reviewer invents blockers. Clean or explicitly contradicted cases make false-positive and contradiction metrics possible.

Stratify the cases instead of reporting one opaque total:

• language and framework;
• diff size and number of changed files;
• local versus cross-file reasoning;
• correctness, security, reliability, compatibility, and release risk;
• tests present, tests missing, and tests that contradict the finding;
• small routine changes versus high-risk migrations or authorization paths.

ReviewGate's manifest uses 30 adjudicated cases as its minimum for an internal regression gate. That is a project convention, not a universal sample-size rule. Choose your corpus size from the uncertainty and slices you need to measure, treat a small corpus's conclusion as narrow, and publish the gaps. Add newly found production defects only after independent adjudication, and freeze them before using them to tune the candidate. Otherwise the test set becomes a training set.

Choose an oracle that verifies the underlying issue

The benchmark needs an answer to “did the reviewer identify this defect?” Text overlap is weak because two comments can describe the same causal failure with no shared phrasing. Location overlap is also incomplete: the useful comment may point to a caller while the human comment points to the callee.

Use the strongest oracle each case supports. Store the expected causal claim, severity, evidence, and adjudication separately from prose. If an LLM judge is needed, freeze its model, prompt, temperature, and matching criteria; retain its decision for audit; and measure agreement on a human-reviewed sample.

Adjudicate contradictions too. A repository test, platform contract, or later code path may prove that a plausible finding is not a defect. Counting those cases separately is more useful than putting every unmatched comment into one generic “false positive” bucket.

Use a scorecard that reflects maintainer cost

Precision and recall are necessary, but a merge gate needs more. Report the raw counts beside every rate so a small denominator cannot masquerade as certainty.

Keep advisory comments in a separate report. A reviewer can offer useful design suggestions while remaining unsafe as a blocking gate. Combining both classes into one “helpfulness” score hides the distinction.

Count duplicate blockers after the first match as noise. Repeating one valid issue across several comments does not improve recall, and it increases the work required to resolve the review. A stable semantic identity also lets you compare repeated runs even when wording and line anchors move.

Report uncertainty, not just point estimates

Publish the numerator, denominator, and an interval estimate for every rate. A Wilson interval is a practical choice for precision, recall, contradiction, and completion proportions; a bootstrap interval can help with per-case counts or latency when its assumptions and resampling method are stated. Do not turn a thin slice into a confident conclusion because its displayed score has two decimals.

Show the number of cases and expected findings in each language, risk class, and diff-size slice. When humans or an LLM judge match findings, report agreement on a blinded sample and document how disagreements were resolved. These disclosures let readers distinguish a stable signal from one case, one judge, or one lucky run.

Measure repeated runs and failed runs explicitly

Model-backed review is stochastic. Run every case at least twice under the same configuration and compare normalized finding identities, blocking status, and completion. More repetitions improve the estimate, but they also raise cost. State the count so readers know what “stable” means in your report.

Use identical inputs for the baseline and candidate. If you are testing a post-processing evidence gate, feed both pipelines the same recorded model response. If you are comparing reviewer agents, hold the model, context budget, repository snapshot, prompt, and tool permissions fixed where possible. Otherwise the report cannot identify which change caused the result.

Failures belong in the denominator. A timeout, provider error, parse failure, or incomplete review is a missed opportunity to catch every expected defect in that case. Dropping failed rows rewards an unreliable reviewer with cleaner metrics. Record the typed failure separately so reliability work has a concrete target.

Resource metrics need the same discipline. Provider-reported spend is stronger than an estimate. When only part of the corpus has latency or cost observations, publish the coverage rather than filling missing values with zero. Report p95 latency when the sample is large enough; a mean can hide a tail that repeatedly stalls pull requests.

A reproducible example from ReviewGate's replay harness

ReviewGate includes a public replay harness for deciding whether one finding policy can replace another. It does not publish review results to GitHub. The current manifest requires a blinded corpus, at least 30 cases, two to ten repetitions, and exactly one baseline plus one candidate. Reports use versioned JSON schemas and include per-case outcomes, threshold decisions, and signed deltas.

On August 6, 2026, we ran the committed deterministic replay at source revision911142b. It loaded 44 cases across seven language/format combinations and six risk classes, with two repetitions. The corpus contains 41 locally maintained policy regression fixtures and three defects with explicit provenance to public pull request #53. The raw-model baseline and evidence-gated candidate consumed the same recorded reviewer outputs. The generated report produced this comparison:

The candidate met every precommitted rollout threshold in this corpus. These are deterministic fixture results from that pinned revision, not observed production accuracy or an estimate of performance on an unseen repository. The point estimates are also not certainty: a 95% Wilson interval for 29/29 has an 88.3% lower bound, while 0/15 contradictions has a 20.4% upper bound.

Anyone can inspect the contract and run the deterministic replay:

cargo run --locked -p reviewgate-cli -- eval-replays \
  --manifest fixtures/evaluation/manifest-v1.json \
  --json-out .reviewgate/benchmark.json \
  --markdown-out .reviewgate/benchmark.md

Theevaluation guide,versioned manifest, andreport schemasexpose the cases, thresholds, metrics, and failure rules. Deterministic mode runs without an OpenRouter key or network call. Live mode is explicit, budget-capped, and non-publishing.

In report schema v1, blocker counts come from the canonical first repetition; later repetitions affect completion and stability. The report includes mean latency but not p95 latency or slice-level confidence intervals. Treat it as an implementation gate that demonstrates the contract, not as a complete example of every analysis recommended in this guide.

The reusable pattern is to treat reviewer changes like any other risky production component. Freeze the inputs, compare one candidate against one baseline, preserve every failure, emit a machine-readable report, and block rollout when a precommitted threshold fails.

Run a repository-specific pilot in seven steps

1. Write the review contract. Name which defects may block, which outputs are advisory, and which pull request head the verdict must describe.
2. Collect historical cases. Sample real defects, representative clean changes, and the highest-cost review failures from your repositories.
3. Adjudicate before testing. Freeze each semantic finding key, severity, evidence, expected outcome, and contradiction evidence with a second maintainer where possible.
4. Blind the subject. Remove fix commits, answer-bearing tests, issue spoilers, and expected findings from the review context.
5. Run baseline and candidate repeatedly. Keep model, context, budget, and repository snapshot fixed; record typed failures and raw counts.
6. Apply thresholds without reinterpretation. A failed rollout can still identify which risk class or context boundary needs work.
7. Shadow production before enforcement. Run the candidate as an advisory check, capture maintainer dispositions, and confirm exact-head behavior before branch protection depends on it.

Themerge-safe GitHub review workflowcovers the event, runtime, evidence, verdict, and freshness bindings that the pilot should verify. ReviewGate'sversioned review artifactsprovide a concrete machine-readable output contract, while theexternal agent workflowshows how a repair loop should consume only a fresh result for the exact head.

Keep the first rollout narrow. One language and one risk class with trustworthy adjudication teaches more than a broad corpus assembled from noisy comments. Expand only after maintainers agree that the benchmark's misses and false blockers match what production review actually costs them.

Interpret benchmark results as bounded evidence

A higher recall score can be worse for the team if it arrives with enough false blockers to train maintainers to ignore the reviewer. Higher precision can also be misleading when the reviewer emits almost nothing. Read recall, precision, false blockers, completion, and stability together.

Compare slices before totals. A reviewer may handle local Python defects and miss cross-file Rust or workflow failures. It may do well on small diffs and collapse on repository-wide migrations. Those gaps tell you where deterministic tests, a specialist review angle, or human approval should remain mandatory.

Public benchmarks are useful for generating a shortlist and challenging vendor claims. Your repository pilot decides enforcement. Preserve the report, corpus version, model and judge identifiers, prompts, context rules, and source revision so a future run can explain a changed result.

ReviewGate is free and open source. You caninstall the GitHub Actionto inspect its score-and-artifact contract, or read thesource and evaluation harnessbefore allowing it anywhere near a merge decision.

AI code review benchmark questions