AI code review on GitHub needs a merge contract.

How should AI code review work on GitHub?

AI code review on GitHub should run as an evidence-producing check, not as an autonomous merge authority. Bind the run to a trusted event and runtime, require exact repository evidence for blockers, derive the verdict from deterministic policy, and accept only a result for the pull request's current head SHA. Human maintainers still decide whether to merge.

Separating defect detection from merge authorization matters because they are different jobs. A model can surface a plausible defect, suggest a test, or point to a risky boundary. It cannot prove by confidence alone that the defect exists, that the repository's policy was applied, or that a later push resolved the problem.

The workflow is easy to remember as five bindings:

1. 1. Bind the review to a trusted event. Use the pull_request event for same-repository pull requests, keep model secrets out of fork and Dependabot runs, and never execute untrusted pull request code in a privileged pull_request_target job.
2. 2. Bind the reviewer to a trusted runtime. Run evidence validation and deterministic scoring from a pinned or attested release that the pull request cannot replace, with only the GitHub token permissions the reviewer needs.
3. 3. Bind every blocker to repository evidence. Require an exact path, diff side, changed line, excerpt, causal path, and test assessment before an AI finding can affect the merge decision.
4. 4. Bind the verdict to deterministic policy. Convert grounded findings into a check result with explicit severity and failure rules instead of letting free-form model prose decide whether a pull request passes.
5. 5. Bind the final result to the current head. Publish a structured result for the exact pull request head SHA, supersede older runs, and let humans merge only after the required current-head checks and review obligations are satisfied.

GitHub makes the same distinction in its native product. Copilot code review submits a COMMENT review rather than APPROVE orREQUEST_CHANGES, so its review does not satisfy required approvals or block merging. GitHub also says Copilot can miss issues or make mistakes and recommends human review. See GitHub'sCopilot code review guideandproduct limitations.

Choose the integration before you tune the prompt

Search results mix four different products under “AI code review”: GitHub's native Copilot reviewer, hosted GitHub Apps, secret-backed Actions, and self-hosted or CI-invoked command-line tools. All can leave useful feedback. Their trust and operating models are not interchangeable.

Use GitHub Copilot when managed operation is the priority

Copilot fits teams that already license it and want a GitHub-native experience. It can gather repository context, follow custom instructions, review new pushes automatically, and offer inline suggestions. Managed operation removes much of the model-key handling, event design, API plumbing, and maintenance that a do-it-yourself reviewer creates. GitHub documents the current context, instruction, automatic-review, and effort controls in itsCopilot code review concepts.

The tradeoff is control. Copilot remains advisory, uses GitHub's supported model and file-coverage contract, and does not become a repository-specific policy engine merely because it reads an instruction file. Choose it when useful comments are the goal. Add a separate required check if the repository needs deterministic enforcement.

Use a GitHub App when fork coverage and central operation matter

A GitHub App can keep the model credential outside a pull request runner and use a narrowly scoped installation token to read changes and publish results. That architecture is often cleaner for organizations reviewing untrusted forks or operating one service across many repositories. In exchange, the service becomes another security, privacy, availability, and billing dependency.

Use an Action or CLI when the repository must own the contract

Choose a CI-owned reviewer when you need provider choice, explicit versioning, machine-readable output, and branch protection tied to a stable check. This pattern carries more security responsibility. A workflow that recovers a model secret by running untrusted code withpull_request_target turns review automation into a credential boundary failure.

If you are still comparing products, start with theAI code review tool decision guide. The rest of this article assumes you chose a repository-owned Action or CLI and need to make it defensible.

Once the integration is chosen, use thePR review prompt libraryto define the finding bar and add focused correctness, security, test, migration, compatibility, frontend, or documentation passes without turning one prompt into a catch-all checklist.

Teams using Claude can compare the local command, managed GitHub review, and a repository-owned gate in theClaude Code review workflow. It separates the author, reviewer, evidence policy, and merge decision instead of treating one model response as approval.

Teams using OpenAI Codex can follow theCodex code review merge-gate workflow to set up@codex review, scope repository rules, and turn advisory P0/P1 findings into a current-head repository verdict.

Teams using Windsurf can apply theWindsurf code review epoch contract to keep Quick Review, pull-request findings, agent autofixes, and the final exact-head gate bound to the code state each one actually inspected.

Teams using Devin Review can use thefour-authority Devin code review contract to separate read, report, branch-mutation, and merge authority while retaining an independent repository verdict for the current head.

Publication needs its own contract. Theguide to pull request review commentsshows how to separate blockers from advice, map findings to line, file, or PR scope, and carry a stable disposition onto the latest head.

What AI code review on GitHub can actually catch

AI review is most useful where a diff contains enough local evidence to test a concrete claim. That includes missing error handling, inconsistent API use, authorization checks that disappear on one path, unsafe input reaching a sensitive operation, and tests that omit a changed failure mode. Repository context can also help connect a changed caller to an existing contract elsewhere in the codebase.

It is less reliable when correctness depends on unstated product intent, production data, undocumented infrastructure, or a large cross-repository interaction the reviewer cannot inspect. A confident comment can still be wrong because the model missed a guard, misunderstood a framework contract, or reviewed only part of a large change. File coverage differs by product too: GitHub documents that Copilot code review excludes dependency-management files, logs, and SVGs.

Keep deterministic tools in their lane. Compilers, type checkers, tests, linters, dependency scanners, SAST, and policy checks provide repeatable answers to narrower questions. An LLM can explain how several signals connect, inspect logic that has no ready-made rule, and propose a missing test. It should not replace a check that already answers the question deterministically.

Noise is an operational failure even when individual comments look reasonable. Set a small blocker budget, suppress style-only findings already covered by tooling, and require the reviewer to state the causal failure rather than merely naming a suspicious pattern. Track which findings maintainers accept, reject, or repeatedly override. That ledger tells you whether the reviewer is learning the repository's risk profile or simply moving human attention from code to bot triage.

The five bindings of a merge-safe AI review

Treat these as independent links. A strong prompt cannot compensate for a stale SHA. Precise line comments cannot compensate for a pull request that can replace the deterministic scoring binary. A green job cannot compensate for a policy that silently turns uncertain prose into a blocker.

1. Event binding: decide which pull requests may receive secrets

For a secret-backed Action, use pull_request and explicitly guard the model-bearing job to same-repository branches. By default—and for public fork pull requests—GitHub withholds Actions secrets and limitsGITHUB_TOKEN to read-only. Dependabot pull requests receive equivalent restrictions. Private-repository administrators can opt into broader fork permissions, which expands the trust boundary; ReviewGate intentionally does not rely on that setting. GitHub'sActions secret guidanceandfork-workflow settingsdocument the boundary.

GitHub documents pull_request_target as a privileged default-branch context suitable for operations such as labeling and commenting. Itssecure-use guidancewarns against checking out and executing untrusted pull request code there. TheOpenSSF Scorecard dangerous-workflow checkindependently flags the same pattern.

2. Runtime binding: isolate deterministic scoring

The pull request should not be able to replace the binary that validates evidence, derives the verdict, or publishes the result. A full-length commit SHA is the only immutable way to pin a third-party Action; moving tags such as ReviewGate's early@v0 channel trade immutability for automatic compatible updates. Run the scorer from a versioned release rather than compiling it from the checkout.

ReviewGate's composite Action downloads a versioned Linux release, verifies its GitHub build provenance, and executes the binary from runner temporary storage. This isolates the attested binary and deterministic scoring implementation; it does not freeze every review input. The workflow, .reviewgate.yml, prompt files, skills, and repository instructions remain governance inputs from the checkout.

ReviewGate's default secret-bearing workflow therefore treats same-repository contributors as trusted to modify those inputs. For adversarial same-repository authors, use an external GitHub App or another base-controlled privileged design. TheAction runtime implementationmakes the narrower binary boundary auditable.

3. Evidence binding: make blockers falsifiable

An actionable blocker needs more than a severity label and fluent explanation. Require an exact path, old or new diff side, one-based line, matching excerpt, and a causal path from the changed code to the user-visible failure. Ask what test covers the claim and demand reproduction-grade evidence for the most severe findings.

Evidence binding changes the review from “the model sounds concerned” to a claim a maintainer can confirm or reject. ReviewGate checks evidence against the actual diff or bounded repository content. Ungrounded candidates remain notes instead of score-affecting findings. The policy is public inthe finding-policy specification.

4. Verdict binding: let code apply policy

The model should propose findings; deterministic code should decide which findings block. Define accepted classifications, severity ceilings, confidence and evidence requirements, verifier behavior, and the difference between review failure and code failure.

ReviewGate allows only P0-P3 defects, security findings, and reliability risks with confidence of at least 0.85 and a passed evidence gate to lower its fixed 5/5 score. Suggestions, contract ambiguity, low-confidence claims, and failed evidence stay advisory. If a review angle fails, the result isreview_error with no score; an unavailable reviewer does not mean the code earned zero.

The repository includes a keyless fixture that demonstrates the lifecycle. The input declares a passing score but also contains a P2 blocker. ReviewGate checks the finding, applies the severity ceiling in deterministic code, and emitsneeds_changes at 3/5 rather than trusting the supplied verdict. The commands and expected output are documented in thelocal score-contract proof.

5. Freshness binding: verify the exact current head

Every published result should carry the repository, workflow, pull request number, and reviewed SHA. Cancel or supersede older runs. Before publication, read the live head again. A result for commit A must not remain authoritative after commit B is pushed.

GitHub'srequired-check rulesrequire a successful result for the latest applicable commit. ReviewGate applies the exact-head contract at artifact retrieval and publication, then exposes the exact SHA in its stableagent result artifact.

These SHAs have different purposes. On a pull_request workflow,GITHUB_SHA normally identifies GitHub's synthetic merge ref, while ReviewGate binds findings to github.event.pull_request.head.sha. Branch protection may separately require the applicable test-merge commit to pass when GitHub has created a status for it. GitHub documents both values in itspull request event reference.

How to set up the workflow

The concrete workflow applies the five bindings in the opening checklist. The maintained same-repository ReviewGate workflow includes the full event guards, permissions, rereview path, checkout settings, and check publication; usethe maintained GitHub Actions guide rather than copying a shortened blog fragment into production.

Keep workflow permissions explicit

Do not start with write-all. GitHub notes that an Action can accessgithub.token even when the workflow does not pass the token directly, andrecommends minimum permissions. A reviewer usually needs read access to repository content plus narrowly scoped write access for its chosen review, comment, status, or check surface. Review the exact integration instead of copying a generic permission block.

Treat pull request text as untrusted data

The diff, title, body, branch name, commit message, and repository instructions can all contain attacker-controlled text. GitHub'ssecure-use referencesays to keep untrusted values out of direct shell interpolation. In the model boundary, label them as untrusted review context rather than instructions with authority over tools, secrets, or policy. A reviewer may analyze a command contained in the diff; it must not execute that command merely because the text asks it to.

ReviewGate enforces a narrower boundary: the model receives bounded review context and returns structured candidate findings. It has no runner shell, GitHub token, or publication tool. Rust code validates the evidence, derives the score, and performs GitHub writes. If your reviewer gives the model tools, document and test a different threat model instead of borrowing this assumption.

Publish one result and one check

Inline comments are useful for local discussion, but they are a poor state database. Prefer one bot-owned summary that is updated across reruns, a structured JSON artifact for automation, and a stable check name for branch protection. This keeps human conversation separate from machine state and gives repair agents a typed input instead of a Markdown-scraping problem.

How AI code review on GitHub should fail

Most setup guides describe the happy path and leave the failure contract implicit. Write it down before making the check required.

• Fork or Dependabot pull request: skip the secret-backed review by policy, then rely on deterministic CI and human review or use a separately designed GitHub App.
• Provider timeout, budget exhaustion, or malformed output: report review unavailable. Do not reinterpret absence of findings as a pass.
• Stale head: discard the result and rerun on the live SHA.
• Finding lacks evidence: retain it as advisory context or suppress it; do not let it affect the required check.
• Comment publication fails: preserve the finding in the structured result and report the publication failure visibly.
• Human rejects a finding: record a reason tied to the current head and evidence, rather than deleting history or training reviewers to ignore the check.

ReviewGate expresses passed, needs-changes, and review-error as separate typed outcomes. Its agent workflow consumes only still-open blockers from a current-head result and stops at a fresh 5/5. It remains review-only: it does not repair code, execute pull request code, or take over the maintainer's merge decision.

Run a pilot that measures signal, not comment volume

Use theAI code review benchmark methodologyto define the corpus, ground truth, metrics, repeated runs, and rollout thresholds before collecting a candidate result.

Do not evaluate a reviewer on a handful of live pull requests where nobody knows the ground truth. Build a small, versioned corpus from historical defects and representative clean changes. Run each candidate against the same cases and repeat enough times to expose drift.

A practical starting scorecard uses at least 30 cases, at least two repeated runs per configuration, exact semantic finding identities, and one denominator that includes timeouts and malformed outputs. Record each expected serious defect before running the candidate. Count a missing or failed review as a recall miss rather than removing the case, and count duplicate blockers as additional noise. These are starting controls, not universal performance thresholds. ReviewGate's pinnedevaluation contractuses those minimums and accounts for failures as misses.

Track precision of blocking findings, recall for serious known defects, false blockers per pull request, duplicate findings, completion rate, rerun stability, latency, model cost, and human override reasons. Keep style advice outside the blocker metric. The business question is whether the reviewer improves merge decisions without consuming more attention than it saves.

ReviewGate uses a networklessreplay evaluationto compare policy changes before rollout. Its corpus is product-specific rather than universal proof, but the method is reproducible: fixed cases, exact expected finding identities, bounded thresholds, and failures counted rather than dropped.

Finally, keep the provenance question in perspective. The same evidence threshold should apply whether a patch was written by a person, an agent, or both. Theprovenance-blind review frameworkshows how to inspect the risky boundaries without pretending coding style proves authorship.

For a concrete policy example, the analysis ofAmazon's disputed AI code review mandateturns a contested provenance story into TRACE risk routing and an auditable exact-head approval record.

Before you make AI review a required GitHub check

A required check changes a reviewer from optional feedback into part of repository availability. Confirm each statement below with a real pull request, not only a workflow read-through.

• The run records the exact PR head SHA and refuses to publish a stale result.
• The pull request cannot replace the binary that validates evidence and derives the score.
• Fork and Dependabot behavior is explicit, tested, and visible to contributors.
• Every blocker includes repository evidence a maintainer can independently check.
• Model or publication failure produces an unavailable result, never a false pass.
• The stable check name is protected against an unintended actor publishing it.
• Maintainers can record an evidence-backed override without erasing history.
• Deterministic CI and required human approvals remain separate from AI review.
• Provider retention, training, regional processing, and source-code egress match policy.
• The team has an owner for action pins, model changes, budgets, and false-positive review.

Also verify product limits against your repository. ReviewGate is early v0 and is intentionally narrower than a hosted review platform: it has no GitHub App, organization dashboard, persistent source storage, automatic repair, or default full-repository index. Its default secret-backed Action path skips fork and Dependabot reviews because GitHub withholds the model secret. Current Action support targets GitHub-hosted Linux X64 runners, and inline publication is best-effort even though findings remain in the structured artifact.

Those constraints may be acceptable for a small same-repository team and wrong for a large public project. If uninterrupted fork coverage, managed administration, or cross-repository context is load-bearing, prefer a carefully scoped GitHub App or native managed reviewer. The correct architecture is the one whose failure modes your maintainers can explain and operate.