# Risk-based code review for AI-written pull requests: a 5-step setup on GitHub

## The short answer

Risk-based code review means you match the review to the risk of the change, not to the size of the diff alone. A documentation fix and a change to payment or auth code are not the same kind of pull request.

[MergeJudge](https://mergejudge.si) is an AI pull request reviewer with an open-source engine. It puts every pull request into one lane: `auto`, `review` or `block`. Auto means low risk. That pull request can go straight through once required checks pass. That is not a guarantee. Review means a person should look. Block means do not merge.

That is the short answer. The rest of this post is an AI code review framework in five steps, and how to set each step up so you can scale code review when AI writes more of the pull requests. Use it when you review AI generated code and you do not want every change to get the same human pass.

The five steps come from Vineet Arya's article "How to Scale Code Review for the AI Era: A 5-Step Framework" (Level Up Coding, 30 Oct 2026): [How to Scale Code Review for the AI Era: A 5-Step Framework](https://levelup.gitconnected.com/how-to-scale-code-review-for-the-ai-era-a-5-step-framework-982ef3a558d3). The steps are his. The setup under each step is written here in our own words.

## Video

The original guide has a slot for a walkthrough video. If one is added, it will be there: https://mergejudge.si/guide/risk-based-code-review-ai

## 1. Match the review to the risk

Not every pull request needs the same review. If you treat a docs typo and a billing change the same way, you either spend too long on the quiet change or not long enough on the dangerous one.

Arya's first step is to put a real risk label on the change, and to let that label drive required reviews and checks. He also says a change that is both large and high risk should be split before anyone tries to read it. MergeJudge does not split a pull request for you. The review comment can name files that could have been their own pull request. A change of more than 5000 lines goes to review.

The label MergeJudge posts is the lane. Make the check `mergejudge/lane` required. How to require it: [stop AI agents merging their own pull requests](https://mergejudge.si/guide/gate-ai-agent-prs#how-to-make-ai-code-review-a-required-status-check). What each lane means: [sort pull requests by risk](https://mergejudge.si/guide/lanes).

| Lane | What it means | `mergejudge/lane` |
|---|---|---|
| `auto` | Low risk. The pull request can go straight through once required checks pass. This is not a guarantee. If branch protection also requires a human review, GitHub still waits for that review. | `success` |
| `review` | A person should look. GitHub treats a neutral required check as a success, so this check does not stop a merge. Keep a required human review. | `neutral` |
| `block` | Do not merge. When the check is required, GitHub stops the merge. | `failure` |

Rules set the lane first. The AI step can only make it stricter, apart from `lanes.code_auto`, which is off by default. By default, code files go to review. Docs, markdown, CSS and `public/**` can be auto when the change is at most 200 lines and nothing else is wrong. One stricter file sets the lane for the whole pull request.

## 2. Tell the reviewer what matters in this repo

A generic review treats every file the same. Your repo has its own rules: what a reviewer must check, and what the linter already handles.

Put the judgement in policy files on the default branch. With no extra config, MergeJudge reads these from the repo root:

- `AGENTS.md`
- `CLAUDE.md`
- `REVIEW_POLICY.md`
- `APPROVAL_POLICY.md`

`.mergejudge.yml` is the config file. It is read from the base branch too. Use it for hard rules, such as paths that always need a person. Use the policy files for judgement, such as "new endpoints must check authorization on the resource".

Files are read from the tip of the base branch, never from the pull request head. A pull request cannot swap in a looser `AGENTS.md` and be reviewed under it. That edit applies from the next review after it is merged. A pull request that edits `.mergejudge.yml` goes to review.

Policy text can raise a lane, from auto to review to block. It cannot lower a lane below what the rules decided. The pull request title, body and diff are untrusted, so a note in the pull request does not replace these files. Each policy file is capped at 8 KB, and 8 KB in total, by default. How the files fit together: [AGENTS.md and policy files](https://mergejudge.si/guide/policy-files-for-ai-review).

## 3. Run special passes you do not have to remember

The third step is to run named passes, instead of hoping someone remembers a checklist. MergeJudge runs three of those as deterministic rules, before any model call. They use `.mergejudge.yml` from the base branch.

Protected paths. Files that match `lanes.protected` are at least review, even when no AI provider is configured. The defaults are `.github/**`, `deploy/**`, `**/auth/**`, `**/migrations/**`, `wrangler.toml`, `_headers`, `**/_headers` and `**/.env*`. `_headers` files stay at least review even if you drop them from the list. If you set `lanes.protected`, you replace the default list, so keep the defaults you still want:

```yaml
# .mergejudge.yml on the default branch
lanes:
  protected:
    - ".github/**"
    - "deploy/**"
    - "**/auth/**"
    - "**/migrations/**"
    - "wrangler.toml"
    - "_headers"
    - "**/_headers"
    - "**/.env*"
```

Secret screen. Added lines that look like an AWS key, a private key block, a GitHub token (`ghp_`) or a hard-coded `api_key` assignment are block. This is a coarse screen, not a secret scanner. Obvious fakes, such as `EXAMPLE` or `PLACEHOLDER`, are noted and not blocked.

Removed safety guards. In a protected path, removing `--match-head-commit`, `persist-credentials: false`, `set -e`, or `--frozen-lockfile` / `npm ci` is block.

The full rule list is on [sort pull requests by risk](https://mergejudge.si/guide/lanes).

## 4. Review every pull request, and let low-risk ones go through

The fourth step is to make the review a system. A review should run on every pull request, and a low-risk pull request should be able to go through without someone remembering to wave it on.

MergeJudge reviews every pull request it is installed on. Each one gets a lane, a summary comment and the `mergejudge/lane` check.

Approvals are off by default. MergeJudge does not merge pull requests itself. See [Approvals](https://mergejudge.si/guide/approvals).

If you want the auto lane to go straight through once required checks pass, that is GitHub's native auto-merge. Set `approval.mode` to `auto-merge-only`. MergeJudge then asks GitHub to auto-merge an auto-lane pull request only after your branch protection and required checks are met. Turn on Allow auto-merge in the repository settings. Leave both off until you choose this. The auto lane still means low risk. It is not a guarantee.

People who look up how to auto approve low risk pull requests should read it that way. Approvals stay off. `auto-merge-only` does not submit an approval. It turns on GitHub auto-merge for the auto lane only.

AI-written pull requests are still pull requests. To keep agent-marked ones out of the auto lane, set `lanes.trusted_authors` with `agents: untrusted` and `applies_to: all_auto`. The agent marks the pull request with the `mergejudge:agent` label or a `MergeJudge-Actor: agent` line. That setting can only take the auto lane away. Setup: [stop AI agents merging their own pull requests](https://mergejudge.si/guide/gate-ai-agent-prs).

## 5. Learn from reverts and follow-up fixes

The last step is to watch what happens after merge, and to tighten review where a low-risk call was wrong. Do not invent a score for it. Record the outcomes you can actually see, and let a person merge any rule change.

On the hosted GitHub App, learning is on by default (`learning.enabled: true`). Reviewer feedback is recorded. Merges over a lane, reverts and follow-up fixes are recorded too. Learning does not change a lane by itself, and it does not edit your config until a human merges a proposal.

A scheduled run can open one proposal pull request. With enough evidence it edits `.mergejudge.yml` in two ways only:

- a stricter protected path, after auto-lane pull requests in one folder were reverted or quickly fixed at least twice
- a rubric note on an existing check that people keep marking as too strict or a false alarm, at least 3 pull requests and 2 different people. Security checks never get a note

The proposal is labelled `do-not-merge` and `needs-human`. MergeJudge lanes it block and does not approve or merge it. With too little evidence the proposal is a report only and changes no config.

The GitHub Action records checkbox feedback and `/mergejudge learn` notes, and uses those notes for proposal pull requests. The revert and follow-up fix records above are for the hosted App.

## Checklist

1. Require the `mergejudge/lane` check. Also require a human review, because a review lane does not stop a merge on its own.
2. Put judgement in `AGENTS.md`, `CLAUDE.md`, `REVIEW_POLICY.md` or `APPROVAL_POLICY.md` on the default branch.
3. Keep `lanes.protected` on the base branch. If you set the list, copy the defaults you still want.
4. Leave approvals off until you want GitHub auto-merge for the auto lane. Then set `approval.mode` to `auto-merge-only` and turn on Allow auto-merge.
5. If coding agents open pull requests, mark them, and keep them out of the auto lane until you accept that risk.
6. When a proposal pull request appears after reverts or follow-up fixes, read it. Merge it yourself only if you agree.

## Five steps and what to configure

| Step | MergeJudge feature | What you configure |
|---|---|---|
| 1. Match review to risk | Lanes `auto`, `review` and `block`, posted as the check `mergejudge/lane`. Auto means low risk and can go straight through once required checks pass. | Require `mergejudge/lane`. Keep a required human review. See [lanes](https://mergejudge.si/guide/lanes). |
| 2. Context for the reviewer | `AGENTS.md`, `CLAUDE.md`, `REVIEW_POLICY.md`, `APPROVAL_POLICY.md` and `.mergejudge.yml`, read from the base branch. | Write the files on the default branch. A pull request cannot use its own copy. See [policy files](https://mergejudge.si/guide/policy-files-for-ai-review). |
| 3. Special passes | Protected paths are at least review. The coarse secret screen is block. Removed safety guards in a protected path are block. | `lanes.protected` in `.mergejudge.yml` on the base branch. The secret screen and guard checks are already on. |
| 4. Review every pull request, and let low-risk ones through | A review on every installed pull request. Opt in if the auto lane should use GitHub auto-merge. | `approval.mode: auto-merge-only`, and Allow auto-merge in GitHub. Off by default. See [approvals](https://mergejudge.si/guide/approvals). |
| 5. Learn from reverts and follow-up fixes | The hosted App records reverts and follow-up fixes. A proposal can add a protected path. A human merges it. | `learning.enabled` is on by default. Do not merge the proposal until you have read it. |

## FAQ

### What is risk-based code review for AI-written pull requests?

Risk-based code review means the human work follows the risk of the change. A docs fix and a change to auth or billing should not get the same review. MergeJudge is an AI pull request reviewer with an open-source engine. It puts every pull request into auto, review or block. Auto means low risk. The pull request can go straight through once required checks pass. That is not a guarantee.

### How do you scale code review when AI writes more of the pull requests?

Run one AI code review framework on every pull request, and spend human time where the lane is review or block. MergeJudge sorts the pull request before anyone has to read the whole diff. The auto lane is low risk and can go straight through once required checks pass. Branch protection still applies.

### Can a pull request loosen AGENTS.md or .mergejudge.yml for its own review?

No. MergeJudge reads AGENTS.md, CLAUDE.md, REVIEW_POLICY.md, APPROVAL_POLICY.md and .mergejudge.yml from the base branch, not from the pull request head. A pull request that edits .mergejudge.yml goes to review. A policy file edit applies only after it is merged.

### Which review passes run before the model?

Three deterministic passes run first. Protected paths are at least review. Added lines that look like common secrets are block. Removing a safety guard in a protected path is block. These do not need a model. The secret check is a coarse screen, not a secret scanner.

### How do you auto approve low risk pull requests?

Approvals are off by default, and MergeJudge does not merge pull requests. Auto means low risk. The pull request can go straight through once required checks pass. For that lane, set approval.mode to auto-merge-only so GitHub's own auto-merge can run, and turn on Allow auto-merge in the repository. That setting does not submit an approval. The lane is still not a guarantee.

## Related

- [Sort pull requests by risk](https://mergejudge.si/guide/lanes)
- [AGENTS.md and policy files for AI code review](https://mergejudge.si/guide/policy-files-for-ai-review)
- [How to stop AI coding agents from merging their own pull requests](https://mergejudge.si/guide/gate-ai-agent-prs)
- [Approvals](https://mergejudge.si/guide/approvals)

---

This post first appeared on the MergeJudge site. The original guide, which stays up to date with the docs, is here: [Risk-based code review for AI-written PRs on GitHub](https://mergejudge.si/guide/risk-based-code-review-ai).

Credit for the five steps goes to Vineet Arya: [How to Scale Code Review for the AI Era: A 5-Step Framework](https://levelup.gitconnected.com/how-to-scale-code-review-for-the-ai-era-a-5-step-framework-982ef3a558d3).

Disclosure: this post comes from the team behind MergeJudge, which is used here as the worked example.

This article was drafted with AI assistance and edited for accuracy against MergeJudge's public docs.