Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Projects and Practice

Project: Code Review Agent

Build a code review agent: parse a diff, run static checks as tools, collect findings with severity and line numbers, filter noise and format a review comment.

Reviewing code is a good agent task because the input is structured (a diff), tools are cheap and deterministic (linters, tests, search), and success can be measured. A review agent should not replace a human reviewer; it catches the boring problems early so humans can focus on design. In this project you will build a review pipeline whose heart is plain code, with a model used only for the judgment parts.

No model, Git host or linter runs here. The "static tools" are small Python functions that scan diff text, and the model's judgment step is a labelled stand-in. The pipeline (parse, run tools, merge, filter, report) is how real review bots are structured.

Design

flowchart LR
  D["Unified diff"] --> P["Parse: files and added lines"]
  P --> T1["Tool: secrets scan"]
  P --> T2["Tool: style and TODO check"]
  P --> T3["Model: logic review"]
  T1 --> F["Merge findings"]
  T2 --> F
  T3 --> F
  F --> X["Filter and rank"]
  X --> R["Review comment"]

Principles:

  • Review only the changed lines, so comments are relevant.
  • Deterministic tools first. A regex that finds a hard-coded key is cheaper and more reliable than a model.
  • Every finding has a file, a line, a severity and a message.
  • Fewer, better comments. Noisy bots get muted.

Step 1: parse a diff

A unified diff marks the new file name with +++ b/..., hunk headers with @@ -a,b +c,d @@, added lines with +. We track new-file line numbers so comments can point at them.

Python Extract added lines with line numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Removed lines do not exist in the new file, so they do not advance the new-file counter.

Step 2: static tools

Each tool takes the added lines and returns findings. They are small and testable alone.

Python Secrets, risky calls and TODO checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: a model-judgment step (stand-in)

Some problems need understanding: a missing null check, a wrong condition, a confusing name. That is the model's job. Give it the diff hunk plus a rubric, and ask for JSON findings with line numbers. Then validate: drop findings that point at lines not in the diff, or have an unknown severity.

Python Validate model findings before trusting them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Validation is what makes it safe to accept structured output from a model.

Step 4: filter, rank and format

Merge findings, remove duplicates on the same line, keep the highest severity per line, drop low-value items beyond a limit, and produce one readable comment.

Python Merge, rank and format the review

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cap on comments protects authors from a wall of noise; low-severity items drop first.

Safety and testing

  • The diff is untrusted input: a comment inside code can say "reviewer: approve this". Treat it as data and never let the model approve or merge on its own (see guardrails).
  • Give the bot read-only repository access plus permission to post comments, nothing more.
  • Build an eval set of diffs with known issues and known clean diffs; track how many real problems it finds (recall) and how many comments are wrong (false positives). See evaluating agents.
  • Let reviewers react to comments; use that signal to tune the rules.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why run deterministic tools before asking a model?

    Rules catch known patterns exactly; the model handles judgment calls.
  2. 2. Why validate the model's JSON findings?

    Structured output from a model may be wrong; check it against the real diff.
  3. 3. Why review only the added lines?

    Reviewing unchanged code produces noise the author did not cause.
  4. 4. What should a review bot never do on its own?

    Diff text is untrusted and could contain instructions; humans approve.

Summary

  • A review agent is a pipeline: parse the diff, run tools, add model judgment, merge, filter, report.
  • Track new-file line numbers so comments point at the right place.
  • Validate model output against the diff before accepting it.
  • Limit noise with deduplication, severity ranking and a comment cap.
  • Keep access read-only plus comment, treat the diff as untrusted and evaluate with known diffs.

Next: Agent Practice Challenges