Skip to content

Operations

Running Factoryal day to day: the web UI, inspecting and sharing runs, users and access tokens, budgets, configuration, database migrations and tests.

Web UI

The API serves the web UI at /app/, and a landing page at /. Links to the UI's old addresses (/runs/..., /workflows/...) redirect under /app/. The UI has:

  • an overview of what's running and what's waiting for approval
  • runs, each with a live graph of the workflow (or a step list) and its timeline. You can cancel or re-run from there. Pending approvals are shown read-only: they are decided in the workflow's channels (Slack, Jira, the API), which record who decided. A finished run shows its outcome: its PRs' state, CI and reviews, and its retro if it has one.
  • workflows, with a form to start a run
  • the catalog, scorecards, factory config (agents, skills, rules and runners) and spend
  • learning: retros (filtered by cause, workflow or agent), the lessons review queue, and proposals, where a draft can be opened as a PR or rejected (Learning)
  • benchmarks: suites, launching them with an estimate first, and each run's Pareto chart, recommendations and one-click promotion; a finished run's Add to benchmark turns it into a task (Benchmarks)

Sign in with a username and password (see Users), or with an API token: the admin token (FACTORYAL_API_TOKEN) or a viewer token. The browser keeps the session token in localStorage. Read-only users see every page, without the buttons that change things.

The Docker image builds the UI. Without Docker, build it once into the Python package:

cd web && npm ci && npm run build      # writes src/factoryal/web/dist; `factoryal api` serves it
npm run dev                            # or: hot-reloading UI on :5173, proxying /v1 to :8000

FACTORYAL_WEB_DIR points the API at another build. With no build present, / returns 404 and the API works as before.

Run timelines, live events and sharing

  • Timeline. GET /v1/runs/{id}/timeline (or factoryal runs show <id>) lists a run's steps, approvals and child runs in order. Each step shows its model, agent version, overrides, tool calls and cost. Add --json --outputs for the raw data with step outputs.
  • Live events. GET /v1/runs/{id}/events streams the timeline as Server-Sent Events. Each event arrives once, with an id, and end arrives when the run finishes. The stream closes after max_seconds (default 25, which fits behind API Gateway). Reconnecting with Last-Event-ID resumes, which EventSource does on its own.
  • Share links. POST /v1/runs/{id}/share returns a signed link (/v1/shared/<token>) that expires after 24h by default, and at most after FACTORYAL_SHARE_LINK_MAX_HOURS (168). Opening it needs a token with runs:read for that workflow. With FACTORYAL_PUBLIC_SHARE_LINKS=true, new links open without one. Turning that setting off makes every link need a reader again. A share link is never a credential for anything else.

Users

People sign in to the web UI (or POST /v1/auth/login) with a username and password. Each user has one of two roles:

Role Can
admin everything the admin token can: start, cancel and approve runs, connect factories, review memories, manage users
viewer (read only) read everything the web UI shows (read scope), except replay runs and retro evidence, which hold captured prompts; change nothing

There are no users at first. Create the first admin with the admin token, from the CLI:

factoryal users add ana --role admin     # prompts for the password; or pipe it on stdin
factoryal users add bob                  # a viewer
factoryal users list
factoryal users role bob admin
factoryal users passwd bob               # reset a password
factoryal users remove bob

or from Factory → Users (g u), signed in as an admin. The API is GET/POST /v1/users, PATCH/DELETE /v1/users/{username}, GET /v1/me and POST /v1/me/password.

  • Passwords are stored as salted scrypt hashes, 8 characters at least.
  • A sign-in lasts FACTORYAL_SESSION_TTL_SECONDS (12h).
  • The role is checked on every request, so a role change applies at once, and deleting a user signs them out. Changing or resetting a password signs out the user's other sessions.
  • Admins can't change their own role or delete themselves. The admin token always works, so losing every admin user never locks you out.

Access tokens

Token Who holds it Can
Admin (FACTORYAL_API_TOKEN) operators, the CLI, CI jobs everything
Worker (FACTORYAL_WORKER_TOKEN) workers the worker scope: the execution plane's routes (run reports, children, waits, agent and code sessions, pull requests, notifications, scheduled starts, benchmark trials, replays, retros), reading the catalog and writing non-system entities. Not users, tokens, approvals, factories or anything else people do
Run token a worker, while it works for one run the worker scopes, limited to that run (and its child runs and the replays it asked for), for FACTORYAL_RUN_TOKEN_TTL_SECONDS (30 days), and only until its run finishes
User session (POST /v1/auth/login) people in the web UI their role: admin, or read for viewers
Viewer (POST /v1/tokens, admin only) people or scripts that should only watch read, runs:read, runs:read:<workflow> and/or catalog:read
Agent session one agent or code step the agent's scopes, for FACTORYAL_AGENT_TOKEN_TTL_SECONDS (1h)
GitHub installation one coding session push to one repository, for 1h

Integrations (GitHub, Slack, Linear, webhook sources) authenticate with their own request signatures, not with Factoryal tokens.

Worker credentials

Workers never hold the admin token. When the API or the dispatcher starts a run, it mints a run token and attaches it to the run's Temporal workflow as a header. The worker sends that token with every call it makes for the run, so a worker acting for one run can't touch another. Calls that belong to no run, such as scheduled starts and benchmark trials, use the worker token.

  • Workers require FACTORYAL_WORKER_TOKEN. A worker without it stops at startup with FACTORYAL_WORKER_TOKEN is not set; it never falls back to FACTORYAL_API_TOKEN. The API needs the same value to accept it, and refuses to start when it equals FACTORYAL_API_TOKEN (workers would hold every scope) or, on any install but dev, when it's unset or .env.example's placeholder.
  • Run routes take only the run's own token (or the admin token). The worker token alone gets 403 there ("this route takes the run's own token"). The exceptions are reading and cancelling a benchmark trial's run, since the benchmark workflow belongs to no run.

A run token is minted when its run starts and lasts FACTORYAL_RUN_TOKEN_TTL_SECONDS (30 days), meant to be longer than any run: the whole run, all its waits and delays added together. A run that outlives it can't finish normally: its worker then calls with the worker token (it logs run token expired), and every run route answers that with 403 this route takes the run's own token. A step that needs a run route fails once its retries run out (retry, 3 attempts by default). The run's status reports, its closing report included, are dropped after 10 attempts, since reporting is best-effort. Reconcile then closes the run's row from its Temporal execution. Give wait steps and approvals on people timeouts that, added to the rest of the run, stay within the token's lifetime (a wait without one has no limit), or raise FACTORYAL_RUN_TOKEN_TTL_SECONDS. Nothing checks this yet.

A run token also stops working once its run has finished: FACTORYAL_RUN_TOKEN_GRACE_SECONDS (5 minutes) after the run's closing report, which comes after its last cleanup, a cancelled run's included. The API then answers 401 with code run_closed, and the worker fails that activity without retrying it.

The CDK stack creates factoryal/worker-token and gives it to the API and to every worker pool; no worker task definition gets factoryal/api-token. On a single host (Docker Compose), set FACTORYAL_WORKER_TOKEN in .env: every service reads it from there (deploy/lightsail/setup.sh generates one).

The run-token signing secret

Run tokens are signed with their own secret, FACTORYAL_RUN_TOKEN_SECRET, so the dispatcher mints them without holding FACTORYAL_TOKEN_SECRET: that one signs agent, viewer and user-session tokens, and anyone holding it could forge an admin token against the API from anywhere. Each secret verifies only its own kind. A token signed with the run-token secret is never anything but a run token, limited to the worker scopes and to its run, and a run token signed with FACTORYAL_TOKEN_SECRET is refused with 401.

The API and the dispatcher need FACTORYAL_RUN_TOKEN_SECRET (32+ random characters); the dispatcher doesn't get FACTORYAL_TOKEN_SECRET. The CDK stack creates factoryal/run-token-secret, gives the API both secrets, and gives the dispatcher only the run-token one. On a single host, set it in .env.

Refused at startup: - No run-token secret on any install but dev (FACTORYAL_INSTALL_ID): the API and the dispatcher stop with has no run-token secret. A dispatcher that lost it, through a missing grant or a typo in the secret's key, stops instead of signing with something else. Only a dev install without one signs run tokens with FACTORYAL_TOKEN_SECRET. - A run-token secret equal to FACTORYAL_TOKEN_SECRET, which would undo the split. - A public secret on any install but dev: the built-in FACTORYAL_TOKEN_SECRET, or either placeholder from .env.example. Anyone can read those in the repo. The API refuses to start with a public FACTORYAL_TOKEN_SECRET, whatever the run-token secret is, since it verifies every other JWT with it. - A public or empty admin token on any install but dev: the built-in dev-token, .env.example's change-me, or an empty FACTORYAL_API_TOKEN.

The CDK stack sets FACTORYAL_INSTALL_ID on every process. On a single host, set it in .env for these checks to apply.

An empty FACTORYAL_RUN_TOKEN_SECRET= counts as unset.

The API must keep the run-token secret for as long as any run signed with it may still be active. Each run carries the token it was started with for up to FACTORYAL_RUN_TOKEN_TTL_SECONDS (30 days). So:

  • Deploying an API with another run-token secret makes every open run fail with 401 on its next call.
  • Rotating the run-token secret needs the same kind of overlap, and there is no setting for a second run-token secret yet. Changing it fails every active run on its next call, so rotate it only when no run is open, in this order:
  • Stop the dispatcher, so automations and reconcile start no more runs. Keep the API up: workers report through it, so the open runs can't finish without it.
  • Wait until no FactoryalRun workflow is open in Temporal, for example temporal workflow list --query "WorkflowType='FactoryalRun' AND ExecutionStatus='Running'". Empty task queues aren't enough: a run waiting for an approval or an event holds its token without any task queued.
  • Change the secret on the API and the dispatcher, then start the dispatcher again.

The API still starts runs while you wait (manual starts, webhooks, schedules, child runs). Check again right before step 3: a run started after that check carries the old secret and fails.

On a suspected leak, skip the wait: accept that the open runs fail, and rotate.

Payload encryption

Temporal history holds every workflow's inputs and results, every activity's arguments and results, failure messages and the run token header: restricted data (data classification). With FACTORYAL_PAYLOAD_ENCRYPTION=on, the API, the dispatcher and every worker encrypt all of it before it reaches Temporal (headers included), so anyone who can read the history without the install's key, in the Temporal UI, the CLI or an export, sees only ciphertext. The one exception is the SDK's version markers (workflow.patched), which hold only a fixed patch name.

Each payload is sealed with AES-256-GCM under a data key, and the data key is wrapped by the install's key: a KMS key on AWS (FACTORYAL_PAYLOAD_KMS_KEY_ID, which CDK creates and grants), or a local key elsewhere (FACTORYAL_PAYLOAD_LOCAL_KEY, base64 of 32 random bytes, e.g. openssl rand -base64 32). A process asks KMS for a new data key every 5 minutes, not per payload. Every process that talks to Temporal needs the same setting and the same key.

A new AWS stack starts with it on (the CDK default); the setting itself defaults to off for local development and installs without a key. On an install that already runs with it off, turn it on in two deploys, so no process meets a payload it can't read:

  1. FACTORYAL_PAYLOAD_ENCRYPTION=decrypt everywhere: every process can read encrypted payloads, and still writes plaintext.
  2. on everywhere: everything new is encrypted. History written before stays readable, since plaintext payloads pass through.

To turn it off, go back to decrypt and stay there until every workflow started under on has closed (and its history aged out of the namespace's retention, if you still want to read it). Temporal Schedules don't close: a schedule created under on keeps its encrypted arguments. So while in decrypt, delete Factoryal's schedules (temporal schedule list, then temporal schedule delete --schedule-id … for each factoryal-schedule:*); the dispatcher recreates them, in plaintext, on its next reconcile pass (FACTORYAL_RECONCILE_SECONDS, a minute by default). A firing due in that gap is skipped, so do it outside busy schedule times. Then go to off.

Keep what reads the history as long as you keep the history:

  • The key. Without it, the history can't be read or replayed. Changing FACTORYAL_PAYLOAD_KMS_KEY_ID to a new KMS key is fine as long as the processes may still decrypt with the old one: a KMS ciphertext names its own key. A local key can't be changed without losing what it encrypted.
  • The install id. It's bound into every payload (and is the KMS encryption context), so changing FACTORYAL_INSTALL_ID, or the CDK install_id or stack name it defaults to, makes the encrypted history unreadable, just like losing the key.

Reading encrypted history. The API is a codec server: POST /v1/codec/decode (and /encode) decodes with the install's key, for an admin credential only. Payloads that carry the run token come back as [redacted run-token], never decoded.

  • Temporal CLI: temporal workflow show -w <run id> --codec-endpoint https://<api>/v1/codec --codec-auth "Bearer <admin token>".
  • Temporal UI: set the namespace's codec server to https://<api>/v1/codec. The browser calls it directly, so its origin must be in FACTORYAL_CODEC_CORS_ORIGINS (default ["https://cloud.temporal.io"]), and the call needs the admin credential: the UI's "pass the access token" option sends Temporal's own token, which the API doesn't accept, so put a proxy in front that adds the Authorization header, or use the CLI.

Budgets and spend

Budgets are set in a factory's factory.yaml (see factory.md):

budgets:
  "team:payments": { monthly_usd: 500, warn_at: 0.8, on_exceeded: queue }
  • A budget caps a calendar month's (UTC) spend for workflow:<id>, team:<id> (the workflows whose owner is that team), project:<id> or factory:<name> (the runs counted towards that project or factory, see Projects). A project can also set its budget as budget: in its project.yaml. Spend is what runs started that month have cost so far, including what they're still spending. To cap one run instead, set the workflow's max_cost_usd: checked before each llm, agent, code or workflow step, so a run can go over by its last paid step (or several in parallel), and child runs count only against their own cap (see Cost cap per run).
  • When the budget is used up, new runs are skipped (on_exceeded: skip) or queued (queue). Queued runs start on the dispatcher's reconcile pass, once the budget is raised or a new month begins.
  • Crossing warn_at and using the budget up each emit one event a month. A budget_event trigger turns those into runs (e.g. a Slack notify).
  • GET /v1/budgets shows spend and state. Removing a budget from factory.yaml deletes it.
  • GET /v1/spend?group_by=factory|project|workflow|step|team|agent|model|factory_version&since=&until= reports cost, optionally filtered with factory=, project= and workflow= (- selects runs counted towards none). step, agent and model split runs by step; a step is <workflow>/<step path>, with map items and parallel branches added up.
  • Every step attempt that used tokens or money also gets an entry in the spend ledger (cost.spend_ledger), with the run's mode and origin and the four token types, and a step.completed event. Entries are never changed, and a retried report doesn't add one twice.
  • factoryal ledger reconcile checks the ledger against the runs and records what it's missing: a step entry for each finished step a report recorded on its run but not in the ledger, and one gap entry for a finished run whose entries don't add up to its usage. It prints what it added, or spend ledger matches runs. It looks only at runs whose usage and entries disagree, takes one run's lock at a time and is safe to run on a live install, and running it again adds nothing. Deploys don't run it: it's a repair tool for when spend by run and spend by ledger disagree.
  • Budgets, /v1/spend and the projects' monthly spend add up the spend ledger. In /v1/spend, runs counts only runs that spent something (a run without ledger entries cost nothing, and a $0 row is noise in a spend report). A project's runs in /v1/projects still counts every run this month, from the runs themselves. Budgets are read in the API and the dispatcher; workers don't read them.

Configuration

Every setting is a FACTORYAL_* environment variable (or a line in .env). The full list is in src/factoryal/kernel/config.py, and .env.example has a commented starting point. The ones you're most likely to set:

Setting Default Used by Purpose
FACTORYAL_DATABASE_URL local Postgres API, dispatcher postgresql+asyncpg://...
FACTORYAL_API_TOKEN dev-token API, the CLI the admin token; never given to workers. Change it.
FACTORYAL_INSTALL_ID dev API, dispatcher names the install in every event (source, installid); lowercase letters, digits and dashes. CDK sets it to the stack name, lowercased, or -c install_id=...
FACTORYAL_WORKER_TOKEN — (required by workers) API, workers the workers' token (scope worker); a worker without it stops at startup. See Worker credentials
FACTORYAL_RUN_TOKEN_TTL_SECONDS 2592000 (30 days) API, dispatcher how long a run token lasts, counted from the run's start: the whole run, all its waits and delays included, must finish within it (Worker credentials)
FACTORYAL_RUN_TOKEN_GRACE_SECONDS 300 API how long a run token keeps working after its run finished, for calls already in flight
FACTORYAL_TOKEN_SECRET dev value API signs agent, viewer and user session tokens (and, on a dev install without a run-token secret, run tokens); 32+ random characters
FACTORYAL_RUN_TOKEN_SECRET — (required outside dev) API, dispatcher signs run tokens only; 32+ random characters. See the run-token signing secret
FACTORYAL_SESSION_TTL_SECONDS 43200 (12h) API how long a username/password sign-in lasts
FACTORYAL_API_URL http://localhost:8000 workers, CLI where to reach the API
FACTORYAL_TEMPORAL_ADDRESS / _NAMESPACE localhost:7233 / default all Temporal
FACTORYAL_TEMPORAL_API_KEY — all Temporal Cloud (enables TLS)
FACTORYAL_PAYLOAD_ENCRYPTION off (CDK: on) all off, decrypt or on: payload encryption. A new AWS stack sets on
FACTORYAL_PAYLOAD_KMS_KEY_ID / _LOCAL_KEY — all the install's key: a KMS key (CDK sets it), or base64 of 32 bytes outside AWS
FACTORYAL_CODEC_CORS_ORIGINS ["https://cloud.temporal.io"] API where the Temporal UI may call the codec server from
ANTHROPIC_API_KEY — llm, coding workers Claude; coding needs Managed Agents access
FACTORYAL_DEFAULT_MODEL / _EFFORT claude-opus-5-5 / high workers when a step or agent doesn't say
FACTORYAL_AUTO_MIGRATE true API, dispatcher migrate on startup (dev only)
FACTORYAL_CODING_BACKEND managed_agents coding workers or claude_code, codex
FACTORYAL_GITHUB_* — API GitHub App
FACTORYAL_SLACK_*, FACTORYAL_LINEAR_* — API Slack and Linear
FACTORYAL_JIRA_*, FACTORYAL_WEBHOOK_SECRETS — API Jira and webhook sources
FACTORYAL_MEMORY_AUTO_ACCEPT / _TTL_DAYS / _PER_STEP 0.8 / 90 / 10 API agent memory
FACTORYAL_PR_REVISION_LIMIT 3 API PR review revisions
FACTORYAL_PROPOSAL_LIMIT 3 API proposals in flight per factory resource
FACTORYAL_CLUSTER_MIN_RUNS / _WINDOW_DAYS 3 / 14 API retros with one cause and agent that make a cluster, and over how many days
FACTORYAL_DISPATCHER_FUNCTION — API the dispatcher Lambda to wake after a commit that gives it work (The dispatcher on Lambda); the CDK stack sets it
FACTORYAL_DISPATCHER_MINUTE_SCHEDULE / _SCHEDULE_GROUP — / default dispatcher Lambda the every-minute sweep schedule it turns on and off; the CDK stack sets it
FACTORYAL_DATABASE_LAMBDA false API, dispatcher Lambdas one connection per use, with idle timeouts (The dispatcher on Lambda); never on a long-lived process
FACTORYAL_DATABASE_IDLE_IN_TRANSACTION_SECONDS / _IDLE_SESSION_SECONDS 60 / 120 API, dispatcher Lambdas with FACTORYAL_DATABASE_LAMBDA: Postgres ends a transaction (and its locks) or a session left idle this long. Keep the first below the dispatcher's timeout (300) and above the slowest call a dispatch batch makes
FACTORYAL_RECONCILE_PENDING_AFTER_SECONDS / _MAX_AGE_SECONDS 60 / 43200 (12h) dispatcher when reconcile checks a quiet PENDING run, and the age past which one with no execution is failed instead of started (see Reconcile)
FACTORYAL_SECRETS_ARNS — all AWS: load settings from Secrets Manager JSON secrets
FACTORYAL_SCALER_CLUSTER / _TABLE / _POOLS — scaler (_TABLE also API, dispatcher) AWS: the ECS cluster, the kick-hold DynamoDB table, and the pools ([{"queue", "service", "min", "max", "workflow", "cpu", "memory"}]) the scaler manages. The CDK stack sets them
FACTORYAL_SCALER_FUNCTION / _HOLD_SECONDS — / 300 API, dispatcher the scaler Lambda a kick invokes, and how long a kick holds each pool up
FACTORYAL_SCALER_QUIET_SECONDS 300 scaler how long a pool must be quiet before it scales in when the dispatch rate can't be used
FACTORYAL_SCALER_MAX_GAP_SECONDS 90 scaler the longest wait between two quiet ticks before the quiet period starts over; must be between one and two tick intervals (60 and 120, exclusive)
FACTORYAL_SCALER_TRUST_DISPATCH_RATE false scaler scale in as soon as the queue's dispatch rate is 0; only after checking the namespace
FACTORYAL_METRICS_NAMESPACE — scaler the CloudWatch namespace of the scaler's metrics; the CDK stack sets Factoryal. Unset, the scaler publishes none

Values set in the environment take precedence over values loaded from Secrets Manager.

Reconcile

Run status reports are best effort, so every FACTORYAL_RECONCILE_SECONDS (60) the dispatcher checks quiet runs against Temporal (workflow id = run id) and repairs their rows:

  • An active run quiet for 10 minutes whose execution has closed is closed with the execution's result, which frees its concurrency slot.
  • A PENDING run quiet for FACTORYAL_RECONCILE_PENDING_AFTER_SECONDS (60) is checked too. If its execution is running, the row is set to RUNNING (its first report was lost). If it has no execution (a crash between committing the row and starting it), reconcile starts it, through the same path as any start: it gets a run token, and REJECT_DUPLICATE means a dispatcher starting it at the same moment can't start it twice. Cancelling it in that state is accepted (202) and recorded as the run's cancelRequestedAt: reconcile then never starts it, and closes it as CANCELLED ("cancelled before its execution started") once it has been quiet for the same minute, notifying its parent run. If a start was already on the way and its execution appears after all, reconcile cancels that execution on its next pass, without waiting for the run to be quiet. With the dispatcher on Lambda, recording the cancel wakes it, and such a run keeps the minute sweep on until it closes, for 15 minutes at most: an execution that ignores the cancel is then left to the hourly sweep.
  • A PENDING run with no execution that has been PENDING longer than FACTORYAL_RECONCILE_PENDING_MAX_AGE_SECONDS (12 hours) is failed instead, with an error starting flagged: stuck PENDING with no execution, and logged at error level. Temporal only rejects a duplicate start while it still retains the earlier execution, so keep this well below the namespace's retention period (1 day at least on Temporal Cloud).
  • Queued runs that now have a free slot, or whose budget allows them again, are started.

The dispatcher logs each pass that changed something as reconcile: {…}, with these counts:

Key Runs in this pass that were…
closed closed: from their execution's result, as CANCELLED after a cancel before they started, or failed as stuck
started queued and started now that a slot or their budget allows
relaunched PENDING with no execution and started by this pass: their execution exists because of it
relaunch_failed PENDING with no execution whose start failed, so the run was failed (could not start run: …)
already_started PENDING with no execution when checked, but started elsewhere before this pass's start (REJECT_DUPLICATE)
flagged failed as stuck PENDING past the age cap (also counted in closed)
cancelled cancelled before they started, whose execution appeared anyway and was asked to cancel

The dispatcher on Lambda

On AWS (ADR-0006) the dispatcher is a Lambda, factoryal dispatcher --lambda, and nothing else dispatches. It has no loop: it runs when something gives it work. Locally and with docker-compose, factoryal dispatcher polls instead.

  • Wake-ups. After a request commits an outbox event or queues a factory sync, the API invokes the dispatcher asynchronously, before it answers (or once it has finished, if the work was committed later or the request failed). Lambda retries a failed invocation twice, a wake-up or a sweep, and sends one that still fails to an SQS queue, whose event source mapping invokes the dispatcher again with the original event. After 10 deliveries a message moves to a dead-letter queue.
  • Sweeps. EventBridge Scheduler invokes it every hour, and every minute while runs are PENDING, QUEUED or RUNNING or events are left to dispatch (a drain that stopped at its deadline). The dispatcher turns the minute schedule on and off itself. Runs waiting for people don't keep it on, so an idle install, or one where every run waits for an approval, wakes the database once an hour. A sweep reconciles runs and syncs schedules; the hourly one also sweeps replay captures and rechecks closed PRs' findings. Each step fails alone, a failed drain included: the others still run, then the invocation fails. A sweep that only found the dispatch lock busy succeeds: the holder is draining, and a retry would only repeat it.
  • Every invocation runs queued factory syncs and drains the outbox until it is empty, then checks once more before it returns. A wake-up that arrives meanwhile (reserved concurrency 1 makes it wait) finds nothing left to do.
  • Worst case. If the API dies between its commit and the invoke, that event waits for the next sweep: at most a minute while runs are active, an hour when none are.
  • A busy dispatch lock fails the invocation with 503, so Lambda delivers the wake-up again. This happens while something else holds the lock: a dead invocation whose transaction Postgres hasn't ended yet, or a factoryal dispatcher someone runs by hand against the same database. It's logged as dispatch lock busy. The alarm counts only dispatch failed, which a failed drain or a failed sweep step (logged dispatch failed in <step>) writes, and Lambda's own Task timed out and Runtime exited (a crash or out-of-memory kill). A second alarm fires when a message reaches the dead-letter queue.
  • Database connections. The API and the dispatcher Lambdas open one connection per use and keep none (FACTORYAL_DATABASE_LAMBDA), so a frozen Lambda doesn't keep Aurora awake. Postgres ends their transactions idle for FACTORYAL_DATABASE_IDLE_IN_TRANSACTION_SECONDS (60), with the locks they hold, and their sessions idle for FACTORYAL_DATABASE_IDLE_SESSION_SECONDS (120). Only those connections get these timeouts: the migrate task and a polling factoryal dispatcher keep their pools.
  • Deploys. The migrate task runs first, then the API, then the dispatcher: it reads the schema the migration writes and signs run tokens the API must already accept. Nothing polls the database, so Aurora pauses on an idle install between hourly sweeps.

Database migrations

The schema is managed with Alembic migrations, which ship inside the package (src/factoryal/kernel/migrations). The first release's schema is one revision, 0001_initial; each later release adds its migrations on top of it.

factoryal db current                 # where the database is, and the latest revision
factoryal db upgrade                 # migrate to the latest revision
factoryal db upgrade --sql           # print the SQL instead (for review or a DBA)
factoryal db downgrade <revision>    # step back to an earlier revision
factoryal db revision -m "add priority to runs"   # after changing src/factoryal/kernel/models.py
  • Development: the API and the dispatcher migrate on startup (FACTORYAL_AUTO_MIGRATE=true, the default).
  • Production: set FACTORYAL_AUTO_MIGRATE=false and run factoryal db upgrade once per deploy, before rolling out new code, e.g. as a one-off ECS task. Never migrate on Lambda cold starts. The CDK stack does this for you.
  • Data catch-up after a migration: factoryal db upgrade only migrates today and prints database at <revision>. A release whose migration can't do all its data work alone (a backfill too big for one transaction, or data the previous release's instances keep writing during a rolling deploy) adds a step to cli.AFTER_MIGRATE: db upgrade runs it after reaching the latest revision, prints what it caught up, and fails if it fails, so the deploy stops. The dispatcher's auto-migrate runs it too, reports a failure and carries on. That release's notes say so, and the release after removes the step.
  • Safety: migrations take an advisory lock and run in a single transaction (Postgres DDL is transactional), so a failed migration changes nothing.
  • Databases these migrations can't upgrade are refused with a message saying what to do, and nothing changes. One at a revision of the pre-release chain 0001_initial replaced, or with Factoryal's tables but no revision, must be recreated. One at a revision this version doesn't have was migrated by a newer Factoryal (a rollback past a release with a migration): run that version again, or restore a backup taken before its upgrade. (The pre-release chain's first revision was also called 0001_initial; a database left at it isn't caught, so recreate any pre-release database.)
  • Drift: tests/test_migrations.py fails if the models and the migrations disagree, so a model change without a migration can't be merged.
  • Indexes on large tables: migrate.upgrade runs every migration inside one transaction that holds the advisory lock, so a migration can't step out of it to build an index CONCURRENTLY (Alembic's autocommit_block needs to). We chose to keep that guarantee: a migration's plain build blocks writes to its table while it runs. A migration that adds an index to a large table creates it IF NOT EXISTS, and its release notes give the definition, so you can build it yourself first:
CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (...) [WHERE ...];
SELECT indisvalid FROM pg_index WHERE indexrelid = '<name>'::regclass;   -- must be true

IF NOT EXISTS only compares the name, so the migration keeps yours. A failed concurrent build leaves an invalid index behind, which IF NOT EXISTS would also skip: check it before upgrading, and drop and rebuild it if indisvalid is false.

Scaling

Workers are stateless. Run one process per queue in production, so each pool scales on its own backlog.

docker compose up -d --scale worker-llm=4                 # locally
factoryal worker --queues llm                              # one pool per process
factoryal worker --queues workflows,light,llm,coding       # or everything in one (dev)

A step's queue: field can send it to a dedicated pool (e.g. a separate llm-batch pool for low-priority work). On AWS, every pool scales to zero when idle (Deploying to AWS).

Scaling to zero on AWS

Every AWS stack scales its workers to zero (ADR-0006): every pool, workflows and light included, goes down to 0 tasks when idle. The scaler (factoryal.kernel.scaling.scaler, a Lambda run every minute) is the only thing that sets the worker services' desired counts; ECS Application Auto Scaling isn't attached to them. Each tick, per pool, it:

  1. reads the ECS service (desired, running and pending tasks) and counts the tasks that are protected from scale-in. Workers protect themselves while any activity runs (protection.py);
  2. reads the pool's kick hold and quiet_since from the kick-hold DynamoDB table;
  3. reads the queue's stats from Temporal (DescribeTaskQueue with stats: workflow tasks for workflows, activity tasks for the others), just before it decides;
  4. sets the desired count:
  5. never below the floor: the pool's minimum, its protected tasks, and 1 while a kick hold's expires_at is in the future;
  6. out by +1, +3 or +10 tasks as the backlog passes 1, 10 or 50, once the tasks already asked for are running;
  7. in to the floor when the backlog is 0 and nothing was dispatched on the queue in the stats window (tasks_dispatch_rate is 0, an average over about 30 seconds). Any non-zero rate keeps the pool as it is.

The kick. When the API or the dispatcher starts a run, or signals one (an approval, a cancel, an event a wait matched, a child run finishing), it kicks the run's pools: the workflows queue, light (where the interpreter reports) and every step's queue in the workflow version. A kick writes a hold per pool, a DynamoDB item whose expires_at is FACTORYAL_SCALER_HOLD_SECONDS (5 minutes) ahead, and invokes the scaler asynchronously, so a cold run's pools all start at once instead of one scaler tick per pool. A kick that fails is logged and the run goes on; its pools then come up on the scaler's next ticks, about a minute per pool crossed. A child run finishing kicks only the parent's workflows and light pools. The API and the dispatcher kick only when FACTORYAL_SCALER_TABLE is set (the CDK stack sets it; locally nothing scales, so nothing is kicked).

The kick runs on the path that starts or signals a run, so its DynamoDB and Lambda calls time out after 1 second (connect) or 2 seconds (read), with 2 attempts in all, about 6 seconds a call at worst. The kick stops writing holds after the first one that fails (the rest would fail the same way, and the scheduled tick covers those pools), and after a failed write or invoke a process skips that kind of call for 30 seconds. So during an AWS outage a process pays about 12 seconds (one failed write and one failed invoke) once every 30 seconds, on the kick that probes, not on every run start. Every kick writes its holds, but it wakes the scaler only when it brings a pool that no successful invoke from this process covered in the last 10 seconds: a burst of starts on the same pools is picked up by one tick (the scaler runs one tick at a time), and a run that needs another pool still wakes it. An invoke covers only the pools whose hold was written before it, so a pool whose write failed wakes the scaler again on a later kick; while no hold can be written, a process wakes the scaler at most once every 10 seconds per pool (the tick still reads each pool's backlog). The 30-second cooldowns count from the failed call, not from the start of the kick. The hold (FACTORYAL_SCALER_HOLD_SECONDS) must be at least 60 seconds, so it outlasts the next scheduled tick.

The quiet-period rule. When the stats came back but the dispatch rate isn't trusted (FACTORYAL_SCALER_TRUST_DISPATCH_RATE off), a pool scales in only after FACTORYAL_SCALER_QUIET_SECONDS (5 minutes) of ticks that each saw no backlog, no protected task, no live hold and no dispatch. The table keeps when that period started (quiet_since) and the last tick that saw the pool quiet (checked_at). Any activity clears it. If two quiet ticks are more than 90 seconds apart (FACTORYAL_SCALER_MAX_GAP_SECONDS), so a tick in between failed, couldn't read the stats or didn't run, the period starts over: nobody saw what the pool did meanwhile. If the scaler can't record a tick, the pool doesn't scale in on that tick.

The gap has to sit between one and two tick intervals (the tick runs every minute; the setting refuses anything outside 60–120 seconds, exclusive). Above two intervals, a busy tick whose clear failed would go unnoticed and a stale period could let a pool scale in at once. The cost of the tight bound: a tick that starts more than 30 seconds late (a slow cold start of the Lambda) also restarts the period, so the pool stays up longer. It never makes the wait shorter. EventBridge Scheduler runs the tick with no flexible window, so ticks are normally seconds apart from the minute.

Unreadable stats. When DescribeTaskQueue fails (an error, throttling) or answers without stats (a namespace that doesn't report them), the backlog and dispatch are unknown, so the pool keeps its count (it can still be raised to its floor) until the stats can be read again.

Before trusting the dispatch rate, check that the install's Temporal namespace reports it. A missing rate reads as 0 (proto3 floats have no presence), so a namespace that doesn't fill it would look idle. With the scaler's settings and the Temporal API key in the environment, run while some work is being picked up:

factoryal scaler --check-stats   # per pool: hasStats, backlog, dispatchRate

If every pool shows hasStats: true and the busy ones a non-zero dispatchRate, set FACTORYAL_SCALER_TRUST_DISPATCH_RATE=true (CDK: -c scaler_trust_dispatch_rate=true). Until then the scaler runs on the quiet-period rule: pools stay up about 5 minutes after their last task, and the scale-in race at pick-up (ADR-0006) is wider. A pool with hasStats: false can't be scaled on its backlog at all: a tick never lowers its count (see Unreadable stats) and raises it only to its floor (its minimum, its protected tasks, 1 while a kick hold is live). So such a pool is stuck: once a kick raises it to 1 it never goes back to 0, costing a task while idle, and it never goes above its floor under load, so max_tasks has no effect. Scale to zero is the only mode (backlog-metric autoscaling is gone), and a new service starts at 1 task, so on such a namespace every pool sits at 1 from the first deploy. Run the check before an install's first deploy, against the namespace the install will use.

Metrics (namespace Factoryal, every tick): per queue (dimension Queue) DesiredTasks, ProtectedTasks, IdleTasks (running, unprotected, with nothing queued) and QueueBacklog; for the install, DesiredTasks (0 on an idle install; left out of a tick that couldn't read every service) and IdleCostUsd, what that minute's idle tasks cost at Fargate ARM on-demand prices (us-east-1). Idle tasks are counted only on ticks that read the stats, so minutes a pool spends up while its stats can't be read aren't in it. Its monthly Sum is the idle worker compute cost. It doesn't count the rest of an idle install (the dispatcher Lambda, the NAT, Aurora's wake-ups for the hourly sweep, storage), which the cost notes list.

Tests

uv run pytest                                  # the whole suite
uv run pytest tests/test_workflows.py -k approval     # one file, filtered
cd infra && JSII_RUNTIME_PACKAGE_CACHE=disabled uv run --group infra pytest tests   # CDK synth + checks, no AWS

The tests start an embedded Postgres (through pgserver) and Temporal's time-skipping test server, so they need no running services. Set FACTORYAL_TEST_DATABASE_URL to use your own Postgres instead.

The Temporal test server is downloaded from temporal.download on first use. If your network blocks that host, download the test server for your platform from the sdk-java releases (the temporal-test-server_<version>_<os>_<arch> archive: linux_amd64, linux_arm64, macOS_amd64, macOS_arm64, windows_amd64), extract it, make the binary executable (not needed on Windows, where it is temporal-test-server.exe), and point FACTORYAL_TEMPORAL_TEST_SERVER at it. Any recent release works with the pinned temporalio SDK; 1.31.0 is known to pass the suite. FACTORYAL_TEMPORAL_DOWNLOAD_DIR keeps the download in a fixed directory instead, which is how CI caches it.

CI (.github/workflows/ci.yml) runs on every pull request and every push to main: the Python suite, the web typecheck and tests, the infra tests, and ruff.

Ruff lints the Python code (pyflakes, pycodestyle errors and import order, at 120 columns); it doesn't reformat it. Run uv run ruff check before pushing, or uv run ruff check --fix to sort imports. A mechanical change across many files (like the one that introduced ruff) goes in a commit of its own, merged with a merge commit, and its SHA goes in .git-blame-ignore-revs so git blame skips it. To get the same locally:

git config blame.ignoreRevsFile .git-blame-ignore-revs

Module boundaries

The code is being split into bounded contexts (src/factoryal/contexts/, see ADR-0001). tests/test_boundaries.py fails when a module imports another context's code directly. A module belongs to the context whose package it is in (factoryal.contexts.<ctx>); factoryal.kernel is shared by all, factoryal.kernel.api_deps only by the routers, and factoryal.entrypoints wires everything together. When the test fails:

  • A new cross-context import: import it through the other context's facade instead, for example from factoryal.contexts.runs.application import service. Another context's adapters/ and domain/ are internals, also once its code has moved (factoryal.contexts.catalog.adapters.outbox).
  • You removed a cross-context import: run uv run python tests/test_boundaries.py --write-baseline to drop it from tests/boundaries_baseline.txt, and lower MAX_BASELINE in the test to the new line count in the same PR. The baseline only ever shrinks.
  • A new module: put it in its context's package, or in src/factoryal/kernel/ if every context may use it.

Workers must not load database code: importing factoryal.entrypoints.worker may not pull in SQLAlchemy, asyncpg, Alembic, factoryal.kernel.db or factoryal.kernel.models, and a test checks it.