Skip to content

Benchmarks

A benchmark replays an agent's past tasks under several configurations (model, effort, skills, budget, environment) and compares how correct and how costly each one is. Use it to choose a configuration, to promote it as the agent's next version, and to catch regressions every night.

Concepts

Term Meaning
Suite Tasks for one agent, plus defaults (budget_usd, max_iterations, ...).
Task A prompt, a repo, the commit to start from (base_ref) and criteria: the checkable things a correct change does. Optional tags say what kind of task it is (docs, backend, ...). Versioned like workflows.
Launch A suite × configurations × repetitions, with the scorers to apply.
Trial One task under one configuration, once. An ordinary run with its own timeline.
Scorer Grades a trial between 0 and 1: rubric, ci, llm_judge or workflow:<id>.

Run a benchmark

Suites and tasks live in a factory repo, like the rest of the configuration:

# benchmarks/payments/suite.yaml (examples/demo)
title: Payments repositories
agent: builder
defaults: { budget_usd: 5, max_iterations: 3 }

# benchmarks/payments/tasks/add-codeowners.yaml
repo: billing-api
base_ref: 4f2a9c1e7d3b             # every trial starts from this commit
prompt: Add a CODEOWNERS file assigning billing-api to the payments team.
tags: [config]                     # results are also broken down by tag
criteria: |
  - .github/CODEOWNERS exists
  - it assigns every path to @acme/payments
  - no other files are changed

kind: BenchmarkLaunch           # examples/benchmarks/launch.yaml
suite: payments
configs:
  - { id: opus-high, model: claude-opus-5-5, effort: high }
  - { id: sonnet, model: claude-sonnet-5-5, budget_usd: 3, max_parallel: 8 }
  - { id: new-prompt, factory_ref: smaller-diffs }   # the agent as it is on a factory branch
repetitions: 2
scorers: [rubric, ci]

factoryal bench run examples/benchmarks/launch.yaml --estimate   # trial count and cost cap only
factoryal bench run examples/benchmarks/launch.yaml
factoryal bench results <benchmark-run-id>

Configurations pinned to a factory ref. factory_ref (a branch or a commit) runs the agent and its skills as they are at that ref of the factory repo that owns the agent (or the suite), instead of what is live. Use it to benchmark a change to an agent's prompt or skills before merging it. The launch reads the factory at the ref and freezes the agent and its skills in the benchmark. Their version is the ref's commit, and results show the config's factoryRef: {factory, ref, sha}. A skill comes from the ref only if it is new or belongs to that factory; others (another factory's, or one the ref doesn't define) are the live ones. Because the ref isn't merged, the agent must already be live, and the ref can't widen its scopes; trials also never get more scopes than the live agent holds when they start. The launch fails if the ref has no such agent, widens its scopes, or uses a skill that is neither in the ref nor live. Estimates (estimate_only) don't read the repo. Rules and lessons still come from what is live, as for the other configurations.

Tasks from past work. POST /v1/benchmarks/<suite>/tasks/from-run with {run_id} drafts a task from a run whose code step opened a PR. The draft has the task text, the PR's base commit as base_ref, and the PR's final diff as the reference change. Write its criteria, then POST it to /v1/benchmarks/<suite>/tasks: that opens a pull request adding benchmarks/<suite>/tasks/<id>.yaml next to the suite's file in the factory repo that owns it. Merging it adds the task, like any other factory change. The web UI does both from a run's page (Add to benchmark).

How trials run

sequenceDiagram
    autonumber
    actor User
    participant API
    participant B as FactoryalBenchmark<br/>(Temporal)
    participant R as trial runs<br/>bench.HASH workflow
    participant S as scorers
    participant GH as GitHub CI

    User->>API: POST /v1/benchmarks/payments/runs (launch)
    API->>API: freeze tasks, configs, agent and skill versions
    API->>B: start benchmark
    loop each task × config × repetition
        B->>R: start trial run (queued by max_parallel)
        R->>R: code step, mode dry_run, base_ref<br/>branch factoryal-bench/BENCH/TRIAL
        R-->>B: diff, cost, outcome
        B->>S: rubric · llm_judge · workflow:ID
    end
    GH-->>API: check_suite for the trial's final commit (ci scorer)
    B->>API: results, compared with the baseline
  • Each configuration gets a generated workflow (bench.<hash>) with one dry-run code step. max_parallel is that workflow's concurrency limit, so extra trials queue without using compute.
  • The launch freezes the tasks, the configurations and the resolved agent and skill versions. If the agent changes while the benchmark runs, its results show agentDrift. A configuration with factory_ref freezes the agent and skills themselves, read from that ref.
  • estimate_only: true (--estimate) returns the trial count and the cost cap (the sum of trial budgets) without starting anything. Launches are limited to 500 trials.
  • Trials never open PRs or write to the catalog, so they don't trigger automations. Webhook handlers ignore pushes to non-default branches.

Scorers

Scorer Scores 1 when Notes
rubric the Managed Agents grader says the change meets the task's criteria (used as the session's rubric) default
ci every check suite GitHub reports for the trial's final commit passes Needs CI that runs on pushes to factoryal-bench/**. Results still pending ci_timeout after the last trial are missing; a late result still updates the score
llm_judge — it scores the share of criteria met, or each of judge.criteria Claude (judge: {model, effort, criteria} on the launch) reads the task, criteria, the trial's diff and the reference change. Its cost counts as scorer cost
workflow:<id> — the workflow returns {score, reason} Any workflow with a trial input (the evidence from GET .../trials/<id>/evidence). Each grading is a run

Judge criteria. judge.criteria makes llm_judge score named criteria one by one, each recorded as llm_judge:<id> on the trial and in each configuration's scores. One judge call grades all of them, and its cost is split between them. Three are built in:

Criterion Scores 1 when
correctness the change meets the task's criteria (the share met)
efficiency the change is as small as the task allows, compared with the reference change, and took few iterations (commits)
verbosity the change has no needless edits: unrelated changes, reformatting, extra files, comments that restate the code

Any other id needs a prompt, the question the judge answers for it. weight (default 1) sets a criterion's share of the judge's part of overall: the criteria count as one scorer, their weighted mean. weight: 0 reports a criterion without counting it.

scorers: [rubric, llm_judge]
judge:
  model: claude-sonnet-5-5
  criteria:
    - { id: correctness, weight: 2 }
    - { id: efficiency }
    - { id: verbosity, weight: 0 }
    - { id: tests, prompt: "Does the change add or update tests for what it changes?" }

Without criteria, llm_judge gives one score, as before. Criteria need llm_judge in scorers. When criteria are added to a benchmark that had a plain llm_judge score, the criteria replace it in overall.

A failed trial scores 0. A scorer that fails records an error score, and the benchmark still finishes. Averages only count scored trials, and every average says how many that was (scored/of).

Rescore. POST /v1/benchmark-runs/<id>/rescore reruns the scorers over finished trials without rerunning the agents. By default it only touches what is unscored (missing, error). scorers can add new ones to a finished benchmark, e.g. ["llm_judge"], and judge can add criteria: {"scorers": ["llm_judge"], "judge": {"criteria": [...]}} grades only the listed criteria, and merges them by id into the benchmark's: only the fields given change, so {id: efficiency} (or a custom criterion the benchmark already has) keeps its weight and prompt (and model and effort change only when given). A rescore without criteria grades all of them.

Results, baselines and promotion

GET /v1/benchmark-runs/<id>/results (or factoryal bench results <id>) returns:

  • a table per configuration: overall score, per-scorer means, average cost and duration, and trials that succeeded
  • a task × configuration grid
  • recommendations: best quality (highest overall), most efficient (overall per dollar) and lowest cost. comparable is false when configurations have different shares of trials scored.
  • frontier: the Pareto-optimal configurations for ?x= and ?y= (default avgCostUsd against overall; also scores.<scorer>, avgDurationSeconds)
  • byTag: the same table and recommendations for each task tag, as {tag: {tasks, configs, recommendations}}. Untagged tasks are under -, and a task with several tags counts for each. - is reserved: a factory whose task is tagged - fails factory check and apply. A task saved earlier with it still loads and re-applies unchanged; its - counts only when it has no other tag. Use it to find a cheaper configuration that is as good on one kind of task. factoryal bench results <id> --by-tag prints a table per tag.

PUT /v1/benchmarks/<suite>/baseline pins a run. Later results then show each configuration's change against it (or pass ?baseline=<id>, or ?baseline=none to compare with nothing).

Promote. POST /v1/benchmark-runs/<id>/promote with {config} opens a pull request on the repo of the factory that owns the agent. The PR sets the configuration's model, effort and skills in the agent's file, and its description has the benchmark's results. Reviewing and merging it is the approval: the merge applies it, and the merge commit is the agent's new version. The agent must belong to a factory in a GitHub repo, and rewriting the file drops its YAML comments. With {config, tag}, the PR quotes that tag's results next to the overall ones and records the tag in its description. For now merging it still changes the agent for all its tasks, and the PR says so. Step model routing will use the tag to add a routing rule instead. A configuration with factory_ref can't be promoted: what it measured is the agent at that ref, so merge the ref instead.

Tasks from runs. POST /v1/benchmarks/<suite>/tasks/from-run turns a past run into a task draft (repo, base commit, prompt). Add criteria and POST it to /v1/benchmarks/<suite>/tasks, which opens a PR adding it to benchmarks/<suite>/tasks/ in the factory. The suite must belong to a factory in a GitHub repo; an identifier that already has a file there is refused (409), so pick another one.

Regressions

A benchmark step runs a suite from a workflow and waits for it. Without configs, it runs the suite's agent as it is now. Put it on a schedule trigger to rerun nightly:

triggers: [{ type: schedule, cron: "0 2 * * *" }]
steps:
  - { id: bench, type: benchmark, suite: payments, scorers: [rubric, ci] }

When a run finishes, it is compared with the suite's baseline. If a configuration's score drops by more than tolerance.overall (default 0.05), or its cost rises by more than tolerance.cost (default 25%), the run records a regression and emits benchmark.regressed. A benchmark_event trigger turns that into a run (e.g. a Slack notify), and /v1/config/report flags the agent until a later run is back within tolerance. See examples/demo/workflows/nightly-benchmark.yaml.

In the web UI

Benchmarks in the navigation (g b) lists the suites and recent benchmark runs.

  • A suite (/benchmarks/<suite>) shows its agent, defaults, tolerance and baseline, its tasks (click one for its prompt and criteria) and its runs. Launch opens the launch form: configurations (id, model, effort, skills, budget, backend; empty fields keep the agent as it is), repetitions, scorers (rubric, ci, llm_judge, and workflow ids) and which tasks to run. It always estimates first, showing the trial count and the cost cap, and any change to the form asks for a new estimate before Launch N trials.
  • A benchmark run (/benchmark-runs/<id>) refreshes while trials run or scores are pending. It has the recommendations (best quality, most efficient, lowest cost), a Pareto chart, the configuration table, the task × configuration grid and every trial.
  • The chart plots y against x (pickers for overall, each scorer, average cost and average time; the URL's ?x=&y= are the API's). Configs on the frontier are orange squares joined by the trade-off line; arrows on the axes say which way is better. Click a point to pick that config.
  • The table has overall and per-scorer means with how many trials were scored, average cost and time, and trials that succeeded. Changes against the baseline (green when better) come from the suite's baseline, or the run picked in compare with (?baseline=).
  • Click a grid cell (or a config) to list only its trials (?task=&config=); each trial links to its run's timeline.
  • Promote (per config and on each recommendation) opens the promotion PR and links to it. Set as baseline, Rescore (unscored trials only) and Cancel act on the whole run.
  • Add to benchmark, on a finished run whose code step opened a PR: pick a suite (and the code step, if there are several), draft the task, edit its identifier, title, tags and prompt, write the criteria, and Open a pull request. The task is in the suite once that PR merges.

API reference

Action API
Suites GET /v1/benchmarks/<suite>, GET /v1/benchmarks (defined in factory repos)
Tasks GET /v1/benchmarks/<suite>/tasks, POST .../tasks/from-run (a draft), POST .../tasks (a PR adding it)
Launch POST /v1/benchmarks/<suite>/runs
Inspect GET /v1/benchmark-runs, GET /v1/benchmark-runs/<id>, .../results, .../trials
Manage POST /v1/benchmark-runs/<id>/cancel, .../rescore, .../promote
Baseline PUT /v1/benchmarks/<suite>/baseline