Durable agent runs
Start, inspect, recover and cancel durable runs using their retained lifecycle receipts.
Q3.1 uses the agent run ID as the persistent execution identity. For long answers, create an agent run in ask mode with durable_temporal: true and response_mode: "receipt". This requires the matching Q3.1 migration, API and Temporal workers. /v1/answer keeps its existing synchronous/streaming execution contract.
{
"prompt": "Answer this question using my authorised resources",
"mode": "ask",
"durable_temporal": true,
"response_mode": "receipt",
"deadline_seconds": 3600
}Send this body to POST /v1/agents/{agent_id}/runs with a unique Idempotency-Key. Save the agent_id, run_id and stream_key receipt before doing anything else. Read GET /v1/agents/{agent_id}/runs/{run_id} to reconcile execution independently of the original HTTP connection. Python and TypeScript support these paths through their generic JSON transport; the new controls do not imply new typed SDK helpers.
States and controls
| State | Meaning / next action |
|---|---|
accepted | Request and dispatch identity are committed; a dispatcher must confirm the Temporal start. |
queued | Temporal accepted the workflow; waiting for a worker. |
running | A worker owns execution and commits checkpoints. |
awaiting_input | Read result_json.pending_input, then submit additional input. An uncertain action instead requires operator reconciliation. |
awaiting_approval | Read result_json.pending_approval and decide the referenced approval using its exact action version. |
completed | Final result is committed in result_json; no more execution. |
failed | Execution or its retries ended unsuccessfully; inspect the safe error code. |
expired | The original execution deadline elapsed; no more execution. |
cancelled | Cancellation is committed; a late worker cannot replace the result with completion. |
Status responses also include durability, lifecycle_version, row_version, deadline_at, started_at and completed_at. The new contract is durability: "temporal", lifecycle_version: 2. Existing inline and version-1 runs are labelled legacy; their old queued/streaming values do not promise restart recovery. The deadline is measured from acceptance (default one hour, maximum 24 hours), including queueing, retries and human waits. Resuming does not extend it.
POST /v1/agents/{agent_id}/runs/{run_id}/cancelpersists cancellation and returns{"cancelled": true}when cancelled. A previously completed/failed/expired run returns false. Notification to Temporal is best effort; the worker observes persisted terminal status as well.POST /v1/agents/{agent_id}/runs/{run_id}/resumeaccepts{"additional_prompt": "The requested information"}with a new idempotency key. It keeps the run ID and deadline, advances the internal workflow generation and returns the same receipt shape. Active and terminal runs return 409. An accepted run can retry dispatch with an empty body; it cannot accept new input until it reachesawaiting_input.POST /v1/agents/{agent_id}/runs/{run_id}/tool-approval/{approval_id}accepts{"approved": true, "action_version": "<returned version>"}. Other approval kinds use the existing approval routes. Decisions remain bound to the same tenant, principal, run and action version.
Run reads and controls require the original principal and tenant. Input resume revalidates workspace and data access. Stream keys are bound to their run. Internal authorisation grants are excluded from result projections. Legacy checkpoint rollback/resume/branch, manual checkpoint insertion, pause and handoff controls cannot be used to bypass the durable lifecycle.
Recovery and limits
Acceptance is committed before dispatch. If dispatch confirmation fails, HTTP 503 returns durable_dispatch_unknown, retryable: false and the saved receipt. Keep that receipt. Query the run or retry dispatch on its existing run ID; do not start a replacement run. An uncertain start is retried using the same Temporal workflow ID with duplicate reuse rejected.
A worker commits checkpoints independently of the final result. PostgreSQL session locks prevent overlapping activity owners; row versions and a database transition guard fence late writes. Retry restores the last committed checkpoint. Terminal activity redelivery returns the saved outcome without running the agent again. Repeated activity failures are finalised as failed; deadline reconciliation produces expired even when no worker ran.
Q3.3 adds a durable tool invocation journal. A committed result can restore a pre-tool checkpoint. An unresolved local intent or uninstrumented action remains at awaiting_input with error_message: "side_effect_outcome_unknown"; normal resume returns run_reconciliation_required. Developer-hosted tools wait with tool_result_required, accept validated results and resume explicitly after all results settle. Unknown external outcomes require inspection and, where appropriate, explicit compensation in the owning system before cancelling the blocked Q run. Automatic cloud PR finalisation remains disabled for managed runs.
This is checkpoint recovery, not exactly-once external execution. HTTP/SSE disconnection does not cancel durable execution, and stream EOF does not establish success. Read the run record for the outcome. Q3.2 adds the shared durable event journal, with authorised cross-instance replay and terminal reconciliation. Inline and old version-1 streams remain best effort. Cancellation/deadline fencing does not undo external actions already sent; Q3.4 adds shared cancellation, resource cleanup and observed usage.
Operator rollout
- Drain old agent activities, apply
20261007_230000_run_lifecycle, and deploy the matching API/workers. The migration aligns agent enum storage with the original lowercase VARCHAR schema. Keep version-1 workflow registration for existing histories. - Give workers a direct PostgreSQL or session-pooling connection. Transaction-pooling PgBouncer is unsupported: activity ownership depends on a session-held advisory lock across commits. Size the database connection budget for concurrent activities plus ordinary requests and observers.
AGENT_CONCURRENCY_LIMITbounds tenant execution slots, including approval waits. - Set
DURABLE_AGENT_RECONCILE_TENANTS='["authorised-tenant"]'on the worker entry point (python -m app.workflows.worker). Its five-second sweep confirms accepted dispatches and expires overdue runs, up to 100 candidates per tenant per sweep. Q3.2 also prunes up to 100 expired event streams per assigned tenant. The default empty list runs no sweep; provisioning must assign all tenants. Multiple reconcilers are safe. A one-shot operator sweep ispython -m scripts.agent_runs --tenant authorised-tenantusing that environment's configured database and Temporal service. - Monitor accepted age, worker availability, deadline failures and reconciliation warnings. Validate tenant assignments, RLS roles and direct connections in staging. Drain all version-2 nonterminal runs before downgrade; downgrade refuses to discard active durable contracts.
Local qualification kills actual API/worker subprocesses against disposable PostgreSQL and Temporal, with a synthetic provider seam. Separate tests exercise the real runner's completion and wait projections, retry exhaustion, cancellation, expiry, database fencing and restricted-role checkpoint commits. This does not qualify live-provider quality, production throughput or hosted rollout.