Skip to main content

The Control Plane — planning, lease queue, and sealed credentials

The Control Plane (cmd/kiwid, pkg/orchestrator) is the SaaS brain: it exposes the API, authenticates callers, runs the Planner, holds the lease queue, and stores sealed credentials. It never runs your code — that is the Data Plane's job.

Roles​

The Control Plane binary runs as one of four roles so the request-serving surface and the background engine can scale and deploy independently:

RoleWhat it does
apiServes the HTTP API and auth only.
orchestratorRuns the background engine and singleton sweepers only.
migrateApplies database migrations and exits. Run before rolling serving instances.
allEverything in one process — convenient for local and single-box dev.

Health checks are exposed at /healthz (liveness) and /readyz (DB-checked readiness). In a multi-replica deployment, run kiwid -role migrate once before serving and set KIWI_SKIP_BOOT_MIGRATE=true on the serving roles so they don't each attempt migrations on boot.

The API surface​

The API is the single entry point for every surface — the kiwi CLI, the Node/Python SDKs, the Next.js dashboard, and the Linear webhook. Notable endpoints:

  • POST /api/v1/planner/plan — submit a task for planning. Supports idempotent submission via the Idempotency-Key header.
  • POST /api/v1/daemon/join-token — mint a single-use join token for a new daemon.
  • POST /api/v1/webhooks/linear — receive Linear issues labeled kiwi (or moved to In Progress) and convert them into planner jobs.

Daemons never receive an inbound connection; they poll the API over HTTPS (heartbeat) and receive leased work in the response. See The Data Plane daemon.

The lease queue​

The lease queue (pkg/store/queue.go) is the heart of the Control Plane's hand-off to daemons. Its defining property: tasks are leased, not popped.

A task is never removed when handed out. It is leased to one daemon for a bounded window (a TTL). If that daemon dies without renewing the lease, the lease expires and the task returns to QUEUED so another daemon can pick it up. This gives crash recovery for free — a dead daemon's work is never lost.

State machine​

Fencing and poison pills​

  • LeaseID is a fencing token. Every renew and every complete must present the current lease id. A stale daemon whose lease has since been reassigned cannot mutate the task — its writes are rejected.
  • MaxLeaseAttempts (5) dead-letters poison pills. A worker spec that reliably crashes whichever daemon leases it would otherwise be requeued forever. After the attempt bound is reached, an expired lease is marked FAILED (dead-lettered) instead of requeued.
  • StartedAt is set once at lease and never touched by renewals, so it is the correct basis for agent-minute metering (unlike UpdatedAt, which resets on every renew).

Fleet scoping and caps​

Every task carries a fleet_id. A daemon leases only work for its own fleet plus unassigned work (fleet_id = ""). At lease time the queue also enforces per-org limits — a concurrency cap, per-job and per-month budget caps, and a monthly agent-minute ceiling — refusing a new lease when a cap is hit. On Postgres the candidate scan uses SELECT … FOR UPDATE SKIP LOCKED, so concurrent daemons never contend for the same row.

DAG gating​

The queue is also where the DAG is enforced: LeaseNextTask only hands out a task once all of its depends_on siblings have SUCCEEDED, and fails a task fast if any dependency has FAILED. That logic is covered in detail on the DAGs page.