Hari KancharlaAI research engineer
← All projectsActive prototype

AI evaluation

Code Review Arena

A benchmark that checks the patch, not just the critique.

Code Review Arena scores bug detection and repair separately, then applies each patch and runs the required tests and validators.

Try demoView repositoryDocumentationBenchmark packs
My role
Independent project · product, benchmark design, backend, and dashboard
Status
Runs locally with deterministic controls and adapters for external reviewers.
Stack
Python · FastAPI · Pydantic · Typer · Docker

The problem

Finding the bug is only half the job.

Code-review benchmarks often stop after bug detection. That makes a convincing explanation look successful even when the proposed patch does not apply or breaks the test suite.

What I built

A benchmark that executes the patch.

I built the benchmark runner, scoring pipeline, isolated patch execution, versioned case packs, API, and dashboard. Detection and repair receive separate scores.

How it works

The main path.

  1. 01

    Give the reviewer a bounded diff and file set without the seeded answer or pre-patch test output.

  2. 02

    Score the reported finding independently from the proposed repair.

  3. 03

    Apply the patch in an isolated workspace with protected paths.

  4. 04

    Run the required tests and structural validators.

  5. 05

    Save the pack checksum, configuration, timings, and per-case result in the run record.

Key engineering decisions

Three decisions that matter.

Separate detection from repair

A reviewer can receive credit for finding the defect without receiving credit for a patch that does not apply or pass the required checks.

arena/scoring/ · arena/benchmark/benchmark_runner.py

Protect the benchmark

Reviewer payloads omit answer-bearing metadata, while protected paths and pack hashes make obvious benchmark tampering visible.

arena/benchmark/ · arena/patching/ · arena/execution/

Keep every result inspectable

Reports retain the harness revision, pack checksum, reviewer configuration, budgets, timings, and repair diagnostics.

arena/reports/ · arena/storage/

Results

Selected outcomes.

Benchmarks, packages, and checks from the project.

1.000 / 0.000

detection / validated repair

In the 10-case run, keyword-gamer found every defect but repaired none. The reference patches passed all 10 cases.

Benchmark results ↗
CI

platform and execution checks

The committed workflow exercises the backend, package, dashboard, platform-safety, and Docker execution paths.

GitHub Actions ↗

Interactive demo

Try the main path.

Pick a case and reviewer. Run the patch through tests, validators, and repair scoring.

Controls

Benchmark replay

Saved benchmark run
case_idsecurity_fastapi_multitenant_admin_bypass_001
Detection1.000
Validated repair0.000
  1. 01
    Finding

    Waiting

  2. 02
    Patch apply

    Waiting

  3. 03
    Regression tests

    Waiting

  4. 04
    Structural validator

    Waiting

  5. 05
    Validated repair

    Waiting

Scope

Current constraints.

  • The case packs are curated and small, so results are limited to these benchmark scenarios.
  • Structural validators are case-specific; the required tests remain the primary repair gate.
  • Current adapters cover deterministic controls and external reviewers; there is no production-model leaderboard.