Backlog

Features inspired by Warp (Oz cloud agents, Warp Drive, Factory Benchmarks / WarpBench), adapted to Factoryal. Sizes: S ≤ 2 days, M ≤ 1 week, L > 1 week. Priorities: P0 unblocks other work or is cheap and high-value, P1 next, P2 later.

Status

M1 is implemented: A1, B1, C1, E1 and F1 below. See workflows.md (schedules, workflow steps) and agents.md for how to use them.

M2 is implemented: G1, G2, G3, G4 and G7 (benchmarks.md). Differences from the plan below: - G3 isolation: trials only run a dry-run code step, which gets no catalog token and opens no PR, so there are no catalog events to tag. A trial whose agent writes to the catalog will need that tag once trials can run other step types. - G3 freezing: the registry keeps one version of each agent, so a benchmark records the resolved agent and skill versions at launch and flags agentDrift when a trial ran a different one. It can't pin the older version. - G4 ci comes from check_suite webhooks for the trial's final commit.

M3 is implemented: B2, A2, A3, A4, C2 and C3 (slack-and-linear.md, agents.md and operations.md). Differences from the plan below: - B2 listens to pull_request_review only. Every inline comment belongs to a review, which arrives as one event, so reacting to pull_request_review_comment too would start one revision per comment. - C3 adds viewer tokens (POST /v1/tokens, scopes runs:read[:<workflow>]), since Factoryal had no read-only identities for people before. - C2's stream re-reads the run every second (poll) rather than being pushed; it closes after max_seconds so it works behind API Gateway, and clients resume with Last-Event-ID.

M4 is implemented: G5, G6, G8, G9, D1, D2, E2 and C4 (benchmarks.md, agents.md, operations.md, workflows.md). Differences from the plan below: - E2: parallel + for_each already fanned out at run time; map adds per-item results and on_error: continue that runs every item instead of failing fast. - D2: lessons are written from agent steps (the remember tool). code steps on Managed Agents have no Factoryal tools, so they read lessons but don't write them. - C4: team budgets use a new owner field on workflows; spend counts runs started in the month, including what they're still spending.

M5 is implemented: F2, F3, B3, B4 and G10 (agents.md and benchmarks.md). Every milestone in this backlog is now built. Differences from the plan below: - F2: the coding worker is the container; Claude Code runs as a CLI in a fresh workspace per task, rather than in a container per task. Environment.image is recorded for a future container runner. - F3: the Codex flags weren't checked against a live install; extend them with FACTORYAL_CODEX_ARGS. Codex cost is computed from tokens at configured prices. - B4: Managed Agents uses its own base image. Setup commands are given to the agent in its instructions there, and CLI backends run them before the agent starts. - G10: a benchmark step on a schedule trigger is how suites rerun; regressions are measured against the suite's baseline.

M6: factories as code (added after the backlog, from Warp's factory-definition layout)

Implemented: a factory directory with kinds taken from folders; factoryal factory plan/apply with ownership, adoption, conflicts and pruning; factory versions recorded on applies and stamped on runs and benchmark runs (/v1/spend?group_by=factory_version); generic webhook sources (jira, github, sentry, hmac-sha256, token) with a webhook trigger and entity mapping; runners as environments with a backend. See factory.md.

Then (2026-10-26) factories became git-only: each factory is a folder of a git repo (FACTORYAL_FACTORIES), and the dispatcher syncs it on push webhooks and by polling, with a plan comment and a factoryal/plan status on pull requests and a factoryal/apply status on commits. The configuration write endpoints, factoryal apply and factoryal factory plan/apply were removed. Benchmark promotion opens a PR on the factory repo instead of publishing the agent after an approval. Admins connect each factory's GitHub repo (one factory per repo) on the web UI's Factory page, which checks the repo and can add a starter factory.

M7 completes the Jira flow: jira_comment and jira_transition notify targets, approvals decided by moving the issue (jira: on approval steps, mapped users, approvers respected), and the example factory running the whole loop from ticket to PR in review.

Milestones

Milestone	Items	Outcome
M1: Foundations	A1, B1, C1, E1, F1	Scheduled runs, isolated trial runs, readable run history
M2: Benchmarks MVP	G1, G2, G3, G4, G7	Compare agent configurations on our own past tasks by correctness and cost
M3: Feedback loops and integrations	B2, A2, A3, A4, C2, C3	Agents address PR review comments; Slack and Linear can start runs and get replies
M4: Better agents	G5, G6, G8, G9, D1, D2, E2, C4	Judge scorers, rescoring, promoting the winner, rules, memory, budgets
M5: More runtimes	F2, F3, B3, B4, G10	Other agent harnesses, self-hosted coding, multiple repos, regression benchmarks

A. Triggers and integrations

A1. schedule trigger · P0 · S Workflows can only be started by manual, entity_event and scorecard_event triggers. Add {type: schedule, cron, timezone, inputs} backed by Temporal Schedules. They are created or updated when a workflow version is saved and deleted when the trigger is removed. - Scheduled runs respect concurrency limits (on_limit: skip avoids pile-ups). - run.trigger records the schedule and the fire time. - Example: nightly POST .../scorecards/<id>/evaluate, which removes the "call it on a schedule yourself" note from the README.

A2. Slack inbound · P1 · M · after A4 A Slack app with mentions and a slash command. Each message upserts a slack_message entity (channel, thread, user, text), so existing entity_event triggers start runs. Verify the signing secret, and map Slack users to Factoryal identities for approvals.

A3. Linear inbound · P1 · M A Linear webhook that upserts linear_issue entities (state, labels, assignee). Example workflow: an issue labelled agent is triaged, approved, then fixed by a code step, and the issue is linked to the PR.

A4. Reply-back step · P1 · S A notify step with targets slack_thread, linear_comment and github_comment, so runs report progress and results where they were started. Credentials stay in the control plane, as they do for the GitHub App.

B. Coding loop

B1. Dry-run code step · P0 · S · needed by G3 Add mode: dry_run: push to bench/<run>/<trial> (or skip the push), never open a PR, keep the diff and transcript as run evidence, and accept a base_ref so the repo is mounted at a fixed commit.

B2. PR review feedback loop · P1 · M Handle GitHub pull_request_review and pull_request_review_comment webhooks on PRs opened by Factoryal. Start the code step again on the same branch, with the review comments added to the task and the rubric. Cap the number of revisions per PR and use the existing concurrency key (one revision at a time per PR).

B3. Multiple repos in one code step · P2 · L Mount N repos, each with its own single-repo token, and open one PR per repo with a shared description and links between them.

B4. Environment definitions · P2 · M Catalog entities for a base image plus setup commands (toolchains, dependency install), referenced by code steps and benchmark configurations, so trials start from the same environment.

C. Observability and cost

C1. Run timeline · P0 · M GET /v1/runs/{id}/timeline merges, in order: step start and end, approvals, LLM and agent turns, tool calls, coding session events, and cost per step. Show it in the CLI with factoryal runs show <id>. Done, except coding session events: the control plane has no Anthropic key, so code steps link to their Managed Agents session_id instead of copying the transcript.

C2. Live stream · P1 · M · after C1 GET /v1/runs/{id}/events (SSE) streams timeline entries while a run is active.

C3. Share links · P1 · S · after C1 Signed, expiring links to a run's timeline. Viewing needs a Factoryal identity with read scope on the workflow; no public links by default.

C4. Budgets and spend reports · P1 · M Monthly budgets per workflow and per team: warn at a threshold, then queue or skip new runs when the budget is exhausted. GET /v1/spend groups cost by workflow, agent, model and team.

D. Agent context

D1. Rules · P1 · M A rule registry kind scoped to the org, a team or a repo, with owner and version. Matching rules are added to every agent and code step automatically, and each run records which rule versions it used. /v1/config/report flags rule version drift.

D2. Agent memory · P2 · M A memory blueprint that agents can write to through a scoped tool (e.g. memory:write:<repo>), holding lessons learned, kept with its source run and expiring after a set time. Retrieve memories for the same repo or service at the start of a step. Writes need review or a confidence threshold.

E. Orchestration

E1. workflow step (call another workflow) · P0 · S · needed by G5 Start another workflow as a Temporal child workflow and wait for its output. Its runs count against its own concurrency limits.

E2. map step · P1 · M Fan out over a list computed at runtime (items: "{{ .steps.find.output.repos }}") with max_parallel, then collect the results. Errors follow the existing on_error setting.

F. Agent backends

F1. Model and effort overrides on code steps · P0 · S · needed by G3 Let a step, and later a benchmark configuration, override the registry agent's model, effort and skill versions without publishing a new agent version. Overrides are recorded on the run.

F2. Claude Code container backend · P2 · L A second implementation of the coding backend interface: Claude Code CLI in a container on the coding workers, giving self-hosted workers for customers who can't use Managed Agents. Gives the same result object (summary, outcome, usage, diff).

F3. Codex backend · P2 · L · after F2 Same interface, so benchmarks can compare models from several vendors.

G. Benchmarks (WarpBench-style)

G1. Suites and tasks · P0 · M benchmark_suite (name, description, target agent) and benchmark_task (repo, base commit, prompt, correctness criteria, optional reference diff), defined as YAML like workflows (benchmarks/<suite>/suite.yaml, tasks/<task>.yaml) and versioned.

G2. Task from a run · P0 · S · after G1 POST /v1/benchmarks/{suite}/tasks?from_run=<id>: takes the prompt from the run's inputs, the base commit from the PR's parent, and the merged diff as the reference. The user edits the correctness criteria before saving.

G3. Benchmark runs · P0 · L · after B1, F1, G1 A BenchmarkRun parent workflow takes configurations × tasks × repetitions and starts trials as child workflows in dry-run mode. The launch request returns the trial count and an estimated cost. Concurrency is capped per configuration, and the run can be cancelled. - The suite, task versions and resolved agent, skill and rule versions are frozen on the benchmark run. - Isolation: trial tokens have read-only scopes, and catalog events carry a benchmark tag that the dispatcher ignores.

G4. Rubric and CI scorers · P0 · M · after G3 A scorer interface where each scorer is its own activity returning {score, reason, cost}. First scorers: - rubric: the Managed Agents outcome result. - ci: check results on the trial branch, from the webhooks we already receive.

G5. LLM judge and custom workflow scorers · P1 · M · after E1, G4 - llm_judge: compares the trial diff and transcript against the correctness criteria and the reference diff. - workflow: any Factoryal workflow that returns {score, reason}.

G6. Rescore · P1 · S · after G4 Rerun missing or failed scorers over the stored trial evidence without rerunning the agent. Show "N/M tasks scored" for each configuration.

G7. Results API and CLI · P0 · M · after G4 Endpoints for: - an overall table (scorer averages, overall score, average cost, duration); - a task × configuration grid; - drill-down to trials, linked to their timelines (C1).

Command: factoryal bench results <run>.

G8. Recommendations and baseline comparison · P1 · S · after G7 - The Pareto frontier for any two metrics, e.g. correctness vs. cost. - Picks for best quality, most efficient and lowest cost. - Comparison with a pinned baseline benchmark run, showing the difference for each metric.

G9. Promote the winner · P1 · M · after G8 Turn a recommendation into a proposed registry change (new agent version with the winning model, effort and skills). The change goes through an approval step and then publishes the new version. The benchmark run is recorded as the reason.

G10. Regression benchmarks · P2 · S · after A1, G8 Rerun a suite on a schedule against the current agent versions. Emit a benchmark.regressed event when the overall score drops below the baseline or the cost rises above it, so a workflow can react, and flag the agent in /v1/config/report.

Dependency summary

A1 ──────────────────────────────► G10
B1 ─┐
F1 ─┼─► G3 ─► G4 ─┬─► G6
G1 ─┘  ▲          ├─► G7 ─► G8 ─┬─► G9
G1 ─► G2          │             └─► G10
E1 ───────────────┴─► G5
C1 ─► C2, C3
A4 ─► A2
F2 ─► F3

Out of scope

The terminal itself, voice input and a local agent UI. The web UI (web/) is an admin console for watching runs and approving them; it is not a product UI for teams yet (no per-user sign-in, editing happens through the API and CLI).