Deploying Factoryal to AWS¶

The CDK app in infra/ deploys:

Component	Runs on
API	Lambda (the Factoryal image + Lambda Web Adapter), behind an HTTP API
Dispatcher (automations, factory syncs, reconcile)	Lambda (the same image), woken by the API and swept on a schedule
Workers `workflows`, `light`	ECS Fargate (`light` on Fargate Spot), 0–10 tasks each, scale to zero
Workers `llm`, `coding`	ECS Fargate, 0–20 tasks each, scale to zero
Scaler	Lambda (the same image), outside the VPC, every minute and on every kick: sets the workers' counts
Database	Aurora Serverless v2, Postgres 16 (can pause at 0 ACU)
Temporal	Temporal Cloud
Migrations	a one-off ECS task, run on every deploy before the new code goes live

Everything runs on ARM64 (Graviton) from one image.

flowchart LR
    GH[GitHub · Slack · Linear<br/>Jira webhooks] --> HTTP
    Users[Web UI · CLI · CI] --> HTTP[API Gateway<br/>HTTP API]

    subgraph VPC
        subgraph app[Private subnets with egress]
            API[Lambda<br/>API]
            D[Lambda<br/>dispatcher]
            subgraph ECS[ECS Fargate · ARM64]
                WW[workflows 0–10]
                WL[light 0–10 · Spot]
                WM[llm 0–20]
                WC[coding 0–20]
                MIG[migrate<br/>one-off per deploy]
            end
        end
        subgraph data[Isolated subnets]
            DB[(Aurora Serverless v2<br/>Postgres 16)]
        end
        NAT[NAT gateway]
    end

    HTTP --> API
    API -. wake-up .-> D
    EB[EventBridge Scheduler<br/>sweeps] -.-> D
    WL & WM & WC -. reports, sessions<br/>via ApiUrl .-> HTTP
    API --> DB
    D --> DB
    MIG --> DB
    API & D & WW & WL & WM & WC --> NAT
    NAT --> TC[Temporal Cloud]
    NAT --> ANT[Anthropic API]
    NAT --> GHAPI[GitHub API]
    SM[Secrets Manager<br/>factoryal/*] -.read at startup.-> API & D & WM & WC
    API & D -. kick .-> SC[Lambda<br/>scaler]
    SC -- desired counts --> ECS
    SC -- backlog, stats --> TC

Workers reach the API through its public URL (ApiUrl), like any other client.

Least privilege. Workers can't reach the database. Only the API and the dispatcher (which reads factory repos) can read the GitHub App key, and only the llm and coding workers can read the Anthropic key. Agents get their own short-lived, scoped tokens at runtime.

1. Prerequisites¶

An AWS account, and credentials with permission to deploy CloudFormation (an admin user or role is simplest; the stack creates IAM roles, a VPC, ECS, RDS, Lambda and secrets)
The AWS CLI, logged in (aws login, aws configure or aws configure sso). Check with aws sts get-caller-identity. The deploy goes to the CLI's default region.
Docker, running (CDK builds the image during cdk deploy; bootstrap doesn't need it)
On an x86_64 machine, ARM64 emulation, because the image is built for Graviton: docker run --privileged --rm tonistiigi/binfmt --install arm64. The emulated build is slow the first time (5–15 minutes).
Node.js 18+ (npx aws-cdk), uv and jq
A Temporal Cloud namespace and an API key for it
An Anthropic API key with Managed Agents access
The GitHub App from github-app.md, with its webhook left inactive for now: its URL only exists after the first deploy (step 4)

2. Create the external secrets¶

Secret keys are Factoryal setting names. Create these once per account, in the region you deploy to. All three must exist before the first deploy; the services read them at startup.

aws secretsmanager create-secret --name factoryal/temporal \
  --secret-string '{"temporal_api_key": "<temporal cloud api key>"}'

aws secretsmanager create-secret --name factoryal/anthropic \
  --secret-string '{"anthropic_api_key": "<anthropic key>"}'

aws secretsmanager create-secret --name factoryal/github --secret-string "$(jq -n \
  --arg id "<app id>" --arg secret "<webhook secret>" --rawfile key /root/github-app.pem \
  '{github_app_id: $id, github_webhook_secret: $secret, github_private_key: $key}')"

Webhook secret: you choose it. Generate one with openssl rand -hex 32, and keep it: it also goes in the App's Webhook secret field in step 4.
Private key: copy the App's .pem to the machine you deploy from, outside the repo, e.g. scp <app>.private-key.pem root@<host>:/root/github-app.pem, then chmod 600 it. Delete it once the secret exists; you can generate a new key in the App settings any time.

The stack creates factoryal/api-token, factoryal/worker-token, factoryal/token-secret and factoryal/run-token-secret itself, with random values. The admin token (factoryal/api-token) goes to the API only; every worker pool gets factoryal/worker-token instead, and the dispatcher gets factoryal/run-token-secret and not factoryal/token-secret (Worker credentials).

3. Deploy¶

Run from infra/ (CDK finds the app through infra/cdk.json; elsewhere it fails with "Specify an environment name like 'aws://123456789012/us-east-1'"). Bootstrap runs the app too, so it needs the same -c context as deploy.

cd infra
# once per account/region
npx aws-cdk bootstrap -c temporal_address=<namespace>.<account>.tmprl.cloud:7233 -c temporal_namespace=<namespace>.<account>

# first deploy: database deletion protection off, so a failed deploy can roll back
npx aws-cdk deploy -c temporal_address=<namespace>.<account>.tmprl.cloud:7233 -c temporal_namespace=<namespace>.<account> -c db_protect=false

# every deploy after the first one succeeds: protection on (the default)
npx aws-cdk deploy -c temporal_address=<namespace>.<account>.tmprl.cloud:7233 -c temporal_namespace=<namespace>.<account>

The image also carries the documentation site (served at /docs/). Its links to repository files point at main unless you set DOCS_SOURCE_REF for the deploy, for example to the release tag being deployed: DOCS_SOURCE_REF=v0.2.0 npx aws-cdk deploy .... It's opt-in because a new value rebuilds the image.

Database protection. The Aurora cluster has deletion protection on by default. If a first deploy fails with it on, the rollback can't delete the cluster and the stack ends in DELETE_FAILED/ROLLBACK_FAILED (see Troubleshooting). So make the first deploy with -c db_protect=false, and once it succeeds, deploy again without the flag to turn protection on. Leave the flag off from then on: deploying with db_protect=false turns protection off.

<namespace>.<account> is the namespace ID shown in Temporal Cloud, e.g. factoryal.a1b2c. Keep each command on one line: if a \ line continuation breaks when pasted, CDK runs without the context and the next line fails with bash: -c: command not found.

Deploy shows the IAM and security-group changes and asks for confirmation. The first deploy takes about 20–30 minutes (the image build, Aurora and the NAT gateway).

Optional context: -c db_min_acu=0.5 keeps the database warm (avoids a ~15s resume after it pauses at 0), -c db_max_acu=8, -c nat_gateways=2 (one per AZ, for availability), -c stack_name=FactoryalStaging, -c factories='<FACTORYAL_FACTORIES JSON>' (the factory repos the dispatcher applies; the API and the dispatcher get it), -c scaler_trust_dispatch_rate=true (see How scaling works; only after checking the namespace) and -c nat_instance=true (one t4g.nano fck-nat instance instead of NAT gateways: about $3–4/month instead of about $32 per gateway, in one availability zone, so no egress while it's down. The AMI is looked up when CDK synthesizes, which needs the account and region).

Payload encryption. Every stack has a KMS key for Temporal payload encryption (alias/factoryal/<install id>/payloads, rotated yearly, kept when the stack is deleted). The API, the dispatcher and the workers may generate and decrypt data keys under it, for their own install only; the migration task can't. A new stack encrypts every payload from the start (-c payload_encryption=on, the default). off and decrypt stay for a live install: moving between off and on goes through decrypt, one deploy at a time. See Payload encryption, including how to read encrypted history from the Temporal UI or CLI.

A stack deployed before these defaults (for example an early staging stack) should be destroyed and rebuilt rather than upgraded. Deploying the current code over one changes three things at once: payload encryption goes from off straight to on unless you pass -c payload_encryption=off (or decrypt, the first of the two steps), the ECS dispatcher is deleted, and light moves to Fargate Spot, which CloudFormation may apply by replacing the service. If you must keep one, pass the encryption mode explicitly, run cdk diff first, and deploy when no run is in flight.

Every deploy runs factoryal db upgrade as a one-off ECS task before the API, the dispatcher and the services update. The API deploys before the dispatcher and the workers, so it already accepts the run tokens they send. If the migration fails, the deploy fails and the old version keeps running.

4. After the first deploy¶

The stack outputs ApiUrl, GitHubWebhookUrl and ApiTokenSecret.

In the GitHub App settings, set the webhook URL to GitHubWebhookUrl, set the webhook secret to the one in factoryal/github, and tick Active.
Connect your factory repos: sign in to the web UI at ApiUrl + /app/ with the admin token, open Factories → Connect a repository, and follow the checks (see factory.md). Or set them in the deployment with -c factories='{"payments": {"repo": "acme/payments-factory"}}'. The dispatcher applies each within seconds, and on every merge after that. With the admin token:

export FACTORYAL_API_URL=<ApiUrl>
export FACTORYAL_API_TOKEN=$(aws secretsmanager get-secret-value --secret-id factoryal/api-token \
  --query SecretString --output text | jq -r .api_token)
uv run factoryal factory status

Upgrading a deployment that has configuration from before factories were git-only? See moving an existing deployment to repos.

How scaling works¶

Every pool scales to zero (ADR-0006). The stack deploys a scaler Lambda (outside the VPC, one at a time) run every minute by EventBridge Scheduler, and a kick-hold DynamoDB table (on demand):

Every pool goes to 0 when idle, workflows and light included. The scaler reads each queue's backlog and stats from Temporal Cloud and sets the services' desired counts: out by +1, +3 or +10 tasks as the backlog passes 1, 10 or 50, once the tasks already asked for are running, and in when the queue is quiet. It is the only writer of those counts: the services leave the count out of the template, so a deploy doesn't reset it, and ECS Application Auto Scaling isn't attached to them. The scaler publishes QueueBacklog, DesiredTasks, ProtectedTasks, IdleTasks and the idle cost (IdleCostUsd) to CloudWatch (namespace Factoryal).
Kicks: the API and the dispatcher hold every pool a run uses for about 5 minutes when they start or signal it, and wake the scaler, so a run's first step starts about 30–60s (a Fargate start) after a cold start, plus ~15s if Aurora was paused.
Spot: only light runs on Fargate Spot. A Spot reclaim ignores scale-in protection, so llm and coding, which run long sessions, stay on demand. ECS doesn't fall back to on-demand FARGATE when Spot has no capacity: light (catalog writes, run reports) then can't place a task and runs stall on it without an error until Spot capacity returns. Watch the light service's events (unable to place a task) if runs stop moving.
Deploys: the worker services keep minimumHealthyPercent at 100, so a deploy starts the new tasks before it stops the old ones.
The rules for scaling in, and the check to run before the scaler may trust Temporal's dispatch rate, are in Operations → Scaling to zero.
No killed runs: a worker running activities marks itself protected from scale-in, so ECS only removes idle workers. A long agent loop is never cut off by scale-in.
Graceful shutdown: on SIGTERM (deploys and scale-in), workers stop taking tasks and finish in-flight ones within 110s. Anything still running is retried on another worker: coding sessions reattach to their Managed Agents session, and a CLI session continues the branch an interrupted attempt pushed (agents).

Cost notes¶

Fixed costs: at idle, every worker pool is at 0 tasks and the dispatcher is a Lambda, so the main fixed cost is the NAT gateway (about $32/month). -c nat_instance=true replaces it with an instance of about $3–4/month. The dispatcher and scaler Lambdas' runs, DynamoDB requests and Scheduler invocations cost cents a month. Aurora storage is cheap, and compute is $0 while paused.
Usage costs: Temporal Cloud and Claude are billed per use.

Upgrading a deployment that ran as Forge¶

The project was renamed from Forge to Factoryal, including names that live outside the code. Before deploying the renamed version over an existing one:

Let in-flight runs finish (or cancel them). Workers now register FactoryalRun, FactoryalScheduledStart and FactoryalBenchmark; a ForgeRun still running in Temporal has no worker to continue it.
Keep the stack: the default stack name is now Factoryal, so deploy with -c stack_name=Forge to update the existing stack instead of creating a second one.
Copy the secrets to the new names: forge/anthropic → factoryal/anthropic and forge/github → factoryal/github.
Delete the old schedules: the reconcile only manages factoryal-schedule:*, so remove the forge-schedule:* ones (temporal schedule list, then temporal schedule delete --schedule-id …). The dispatcher recreates them under the new prefix.
Rename settings anywhere they are set by hand: FORGE_* environment variables become FACTORYAL_*, and the CLI is factoryal.
Integrations: the Slack slash command is /factoryal; workflows that read the Slack entity's forge_user property should read factoryal_user. Coding agents now push branches under factoryal/ and tag Managed Agents with factoryal_agent, so existing agent records are recreated on first use.

Troubleshooting¶

missing context: pass -c temporal_address=...: the -c flags didn't reach CDK. Pass them to both bootstrap and deploy, on one line.
Specify an environment name like 'aws://…': you're not in infra/.
Unable to resolve AWS account to use: the AWS CLI isn't logged in. Run aws login (or aws configure) and check with aws sts get-caller-identity.
Cannot use the jsii package cache … writable by group or others: a parent of the jsii cache (often ~/.cache) is group- or world-writable. Fix it with chmod go-w ~/.cache, or set JSII_RUNTIME_PACKAGE_CACHE=disabled.
exec format error while building the image: ARM64 emulation isn't installed (see the prerequisites).
Stack stuck in DELETE_FAILED or ROLLBACK_FAILED on Database… ("Cannot delete protected Cluster"): turn off the cluster's deletion protection, then delete the stack again. The stack takes a final snapshot of the database; delete it in the RDS console if you don't need it.

aws rds modify-db-cluster --db-cluster-identifier <cluster id> --no-deletion-protection --apply-immediately
aws cloudformation delete-stack --stack-name Factoryal

- BlockedException: Your account is currently blocked from ECS RunTask: AWS hasn't enabled Fargate on the account yet (common on new accounts). Check billing and account verification, then open an Account and billing support case asking to unblock ECS/Fargate in your region. It isn't a stack problem: redeploy once the account is unblocked.