Skip to content

Efficiency

The spend report says what steps cost. The efficiency report says whether that money bought good work, so you can see which steps could run on a cheaper model or effort, and which cheap ones cost more than they save.

How steps are scored

Every llm, agent and code step that finished gets a score between 0 and 1 from the best signal it has:

Signal Score
the step failed 0
an approval with reviews: <step> decided it 1 approved, 0 rejected
the inline comments it wrote for a review (findings) the share the author acted on
a pull request the code step opened (outcomes) merged 1 · merged after review asked for revisions 0.7 · closed unmerged 0 · reverted 0
a PR still open, or no signal at all unscored

A step succeeds when it finished and scored at least 0.5, or finished with no score. A step that a goto loop ran again counts once, with the cost of all its attempts.

The report

GET /v1/efficiency?group_by=step&since=&until=&factory=&project=&workflow= (admin), or the web UI's Efficiency page. group_by is step (<workflow>/<step path>), model, agent or route (a step's routing rule). Each row:

Field Meaning
steps, runs, succeeded, successRate how many steps, in how many runs, and how many succeeded
meanScore, scored the mean score of the steps that have one, and how many that is
avgCostUsd, p50CostUsd, p90CostUsd cost per step
costPerSuccessUsd everything the steps cost, failures and retries included, divided by the successes
cacheHitRate prompt tokens read from cache, of all prompt tokens
uncachedInputTokens, cacheReadTokens, cacheWriteTokens the prompt by token type. Each is priced at its own rate (the price list in contexts/runs/application/activities.py): cache reads cost far less than uncached input, and cache writes somewhat more. The web page shows reads and writes under the cache hit rate. Codex steps recorded before this split counted their cached input inside uncachedInputTokens as well as in cacheReadTokens, and priced it as input, so a window that spans the upgrade overstates their input and cost and understates their cache hit rate.
avgAttempts, avgDurationSeconds goto reruns and time per step
variants the same figures split by model and effort (- is the default)

Reading it

  • Compare cost per success, not cost per call. A model at half the price that fails or needs revisions twice as often costs the same per success and wastes reviewers' time.
  • Before trusting a difference, look at scored: a mean over three steps says little. A step with no signal (no PR, no reviewing approval) is only scored by failures; give it an approval with reviews:, or compare models on it with a benchmark.
  • A low cacheHitRate on a step that runs often usually means its prompt puts the parts that change (inputs, earlier outputs) before the parts that don't (instructions, schemas).

Review findings

A reviewing step is good when its comments lead to changes. Give the review's notify (target: github_review) findings_of: <the step that wrote the comments>, and each inline comment it posts is kept as a finding of that step. GitHub's webhooks then decide each one, with nobody asked:

Finding When
addressed a later push changes a line within 3 of the one it points at, or removes or renames its file
disputed the reviewer agrees it was wrong: a github_reply with concedes answered its thread
acknowledged its thread was resolved and no push changed its lines (yet: a later one makes it addressed, and unresolving reopens it). Neutral: "fixed elsewhere" or "won't fix"
stale the commit it was posted on left the PR's history (a force-push or rebase), or at close GitHub's diff leaves its file out (too large, binary, or past 300 files), so its lines can't be followed. Neutral
ignored the PR was merged or closed with it still open (if GitHub failed to show the last push's diff, it stays open and unscored instead: the dispatcher compares it again hourly, and a week after the close settles it as stale)

Reopening the PR reopens the findings its close settled as ignored (or left to recheck).

The step's score is addressed ÷ (addressed + disputed + ignored), once any is decided: a precision for the reviewer, so a cheaper model that flags fewer real problems or more noise shows it. Its row adds findings (counts by state), findingsScore and costPerAddressedFindingUsd. A step that loops (one review per push) scores all its findings in the run together, against the cost of all its attempts.

Needs the GitHub App to send Pull request review thread events for acknowledged (github-app.md); the rest come with Pull request events.

Limits worth knowing: - disputed rests on the reviewer's own judgement of the author's reply, and that reply is untrusted text: an author who pushes back hard enough can talk a reviewer into conceding, and lower its score. Read a low score with many disputed findings with that in mind. - A push that merges the base branch in brings its changes along, so a change from the base branch near a commented line also counts as addressed.

Recommendations

GET /v1/recommendations?since=&factory=&project=&workflow= (admin; default the last 30 days), also on the Efficiency tab of Spend, turns the report into changes to steps. Each has the step, the change, its evidence, a confidence and an estimated monthly saving at the step's current rate.

Kind When Change
cheaper_variant a step ran on several models or efforts, and a cheaper one succeeded as often and scored as well (within 5 points) the cheaper model and effort
costlier_per_success a cheaper variant costs over 10% more per success than a dearer one the dearer model and effort
low_cache_hits a step that runs often reads under 20% of its prompt from cache none: reorder its prompt
try_smaller_model an llm step nearly always succeeds on a top model and never ran on another none: replay it first
replay_cheaper a replay judged a candidate at 0.85 or more against today's answers, at 80% of the cost or less the candidate's model and effort

A model or effort needs at least 10 steps to be compared. POST /v1/recommendations/<id>/apply (the page's Propose button) opens a proposal: a pull request on the workflow's factory repo that sets the step's model and effort, edited in place so the file keeps its comments. Merging it applies the change; the report then shows whether it held up.

Replays

The report can only compare models a step has already run on. A replay tries new ones on the step's real requests, without touching production:

kind: Workflow
identifier: triage-issue
replay: { capture: true, retention_days: 14 }   # keep what llm (and tool-less agent) steps were asked and answered

curl -X POST $API/v1/replays -H "Authorization: Bearer $TOKEN" -d '{
  "step": "triage-issue/classify", "sample": 30,
  "candidates": [{"id": "haiku", "model": "claude-haiku-5-5", "effort": "low"},
                 {"id": "sonnet", "model": "claude-sonnet-5-5", "effort": "medium"}],
  "judge": {"effort": "high"}}'
  • The replay takes the step's latest sample captures and runs them as a generated workflow (replay.<hash>): each candidate answers each request, then a judge scores every answer between 0 and 1 against the one the step gave (1 as good or better). It's an ordinary run, so it shows in Runs and its cost in Spend.
  • GET /v1/replays/<run id> gives each candidate's mean score, failures and cost per answer, against the step's original cost. A failed answer scores 0; one the judge left unscored counts as unjudged, in no mean. Cost counts only answers that came back, so failing doesn't look cheap.
  • A finished replay is evidence for a replay_cheaper recommendation once a candidate has at least 10 judged requests, and the judge scored at least 80% of the requests replayed (the captures found, which can be fewer than sample): with 10 replayed, one request the judge skips rules it out, so use a sample somewhat above 10. A smaller one is a quick look. Each step's newest replay with that many judged requests (for any one candidate) counts, whatever it found, and decides for every candidate it ran: a thinner one after it (a quiet month, many failed answers) doesn't hide it. A candidate that replay judged thinly (its answers failed, or the judge skipped them) isn't recommended on an older replay's evidence: replay it with a larger sample, and if the judge keeps skipping its answers, check them and the judge. A replay its cap stopped fails, so it counts for nothing and hides nothing, but it also throws away the candidates it had already judged, and their cost is spent: size max_cost_usd for every candidate, or drop some. Only replays started in the recommendations window (since, the last 30 days by default) count, so evidence older than that expires.
  • GET /v1/recommendations also returns replays: each step's newest finished replay in the window, recommended or not, with a verdict for each candidate: cheaper (it backs a recommendation), worse (scored under 0.85), not_cheaper (as good, but over 80% of the cost), unknown_cost (as good, but there's no original or candidate cost to compare) or too_few_judged. counted names the replay recommendations go by: this one, an older one when this one judged too few, or null. The Efficiency tab of Spend shows them under Replays.
  • llm steps are captured and replayed: their requests are self-contained and have no side effects. So are agent steps whose agent has no tools (its scopes offer none, e.g. scopes: []): it makes the call an llm step makes, and the capture is the request as sent, its system prompt with skills, rules and the step's memory lessons folded in, with either LLM provider (through the Claude Code CLI there's no max_tokens, as the CLI uses its own limit, and the cost covers the CLI's own turns, e.g. the one that returns the output_schema answer, which a replay through the CLI takes too). A replay asks it again as a plain llm call, and a replay_cheaper recommendation sets the step's model and effort, overriding the agent's for that step (Overrides). An agent with any tool isn't captured: its turns read and write through tools. Compare code steps with benchmarks.
  • Captures hold prompts and answers, which may include whatever the step's inputs held. Each is deleted retention_days (1–90) after it was taken, by the dispatcher's periodic sweep, whether or not the workflow still runs. A replay's run carries the prompts in its inputs, so only admins can read it and it can't be shared. Once the captures it used expire, the sweep scrubs it: inputs, answers and the judge's reasons go; scores, costs and statuses stay. Temporal keeps the run's history for its namespace's retention period.

On a schedule

A replay step does the same from a workflow and waits for the judge, so cheaper models get tried on fresh requests every week with nobody asking:

kind: Workflow
identifier: weekly-replay
triggers:
  - { type: schedule, cron: "0 4 * * 1" }
concurrency: { limit: 1, on_limit: skip }
steps:
  - id: review
    type: replay
    step: pr-review/review
    sample: 20
    since: 7d            # only last week's requests: none is judged (and paid for) twice
    max_cost_usd: 10     # the replay's own cap: it runs as a run of its own
    candidates:
      - { id: sonnet, model: claude-sonnet-5-5, effort: medium }
      - { id: haiku, model: claude-haiku-5-5, effort: low }
    judge: { effort: high }

since (a literal duration, not a template) keeps a quiet week from replaying requests an earlier run already judged; match it to the schedule's interval. max_cost_usd caps the replay run itself: the replay is a run of its own, so the calling workflow's max_cost_usd doesn't cover it. The cap is per replay: a retry of the step finds the replay it already started, but a goto back to the step starts a new replay, with a cap of its own. A replay stopped by its cap fails, and only a finished replay is evidence, so size the cap for a whole run: each request costs one answer per candidate plus one judge call that sees the request, the reference and every answer.

Its output is the replay's summary (each candidate's meanScore and avgCostUsd against the original), with no prompts or answers, so later steps can post it. The replay is evidence for replay_cheaper recommendations like any other. A step with no captures fails, and a retry of the step finds the replay it already started instead of starting another.