Idempotency and retry contract
Persist operation keys and recover uncertain requests without creating duplicate work.
Q1.5 protects keyed POST, PUT, PATCH and DELETE requests under /v1/ and /api/v1/. Send Idempotency-Key with 1–200 visible ASCII characters. X-Idempotency-Key remains an alias for legacy ingestion/report callers. Conflicting header values return 400. Use one random key per intended operation and persist it before sending the request.
Keys are scoped to the authenticated tenant and principal: an API-key record ID, or a verified JWT issuer and subject. Every attempt authenticates again. Keys, principals and request content are represented by hashes in the ownership table; it does not retain bearer tokens, prompts or generic response bodies. Legacy shared-secret authentication, when explicitly enabled, represents one shared service principal per tenant.
Request identity and retention
A key binds method, path, query, body, Content-Type, semantic headers and authority. JSON whitespace and object-key order normalise; arrays and repeated query values retain their order. Semantic headers include Accept, Content-Encoding, If-Match, If-None-Match, Last-Event-ID, X-Tenant-ID, X-User-ID, X-Space-ID, X-Request-ID and Prefer. Keep X-Request-ID stable when retrying a keyed request because existing run APIs use it as an execution identifier. Q6.5 Python SDK builds derive this ID deterministically for keyed calls; older clients must explicitly retain it through the generic transport. For operations started before upgrading, preserve the original request ID rather than substituting the new derived value. Changed scopes or service authority also conflict.
The keyed request body limit is 2 MiB. Invalid/non-finite JSON returns 422 before dispatch; oversized keyed requests return 413. Multipart and other non-JSON bodies bind their exact bytes. File/one-shot request streams are not automatically retried by the maintained SDKs.
Completed operations retain deduplication for 24 hours after finalisation. After that window a key can represent a new execution, so applications should use a fresh key for a new logical operation and should not send delayed retries past the window. In-progress leases last thirty seconds and renew every ten seconds. Expired leases become uncertain and are never automatically reassigned. Unknown outcomes have no automatic expiry.
Responses and recovery
| Outcome | HTTP / metadata | Caller action |
|---|---|---|
| Different request or authority | 409 idempotency_key_conflict | Correct the caller's key/request association; do not blindly retry |
| Original operation running | 409 idempotency_in_progress, X-Q-Retryable: true, Retry-After: 1 | Retry the identical request with the same key within the retention contract |
| Safe creation receipt available | Original receipt status, X-Idempotent-Replayed: true | Use the existing resource identifiers |
| Stream creation already completed | 409 idempotency_stream_reconnect when SSE was requested, identifier receipt included | Use the existing run's GET stream interface subject to its current permission/replay policy |
| Generic mutation completed | 409 idempotency_completed, X-Q-Retryable: false | Read its resource with current permissions; the original body is not cached |
| Execution uncertain | 409 or 503 idempotency_outcome_unknown, X-Q-Retryable: false | Reconcile the operation; neither a new key nor elapsed time proves it is safe to repeat |
| Store unavailable before dispatch | 503 idempotency_store_unavailable, retryable | Retry the identical keyed request after recovery; this attempt did not dispatch |
Claimed operations expose X-Operation-ID; idempotency errors also include error.operation_id and error.retryable. Successful responses use X-Idempotency-Status: protected, stored, replayed or streaming-not-replayable. A server error after dispatch is non-retryable. If storage fails after response headers were sent, the connection may terminate rather than replace the status; retrying the same key still cannot acquire the reserved operation.
Creation receipts are explicitly registered by the server and contain identifiers only. Currently agent run creation uses this feature. Generic answers, searches, resource mutations and legacy ingestion/report creation do not replay private content or a generic success body. The ingestion job ID remains the caller's document ID. Reports receive a server-generated UUID; caller keys are no longer used as job IDs. Resource reads remain subject to Q1.1–Q1.3 permissions. Caller authentication does not by itself authorise replaying old evidence.
Agent creation and streaming
POST /v1/agents/{agent_id}/runs
Authorization: Bearer <credential>
Idempotency-Key: <persisted-operation-key>
Accept: application/json
Content-Type: application/json
{"prompt":"Inspect the project","workspace_id":"<workspace-id>","response_mode":"receipt"}The 201 body contains agent_id, run_id and stream_key. Retry the same creation request/key to recover that receipt. Both Python q.agents.runs.start(...) and TypeScript q.agents.runs.start(...) now request it. Then use the existing GET /v1/agents/{agent_id}/runs/{run_id}/stream?stream_key=... interface. POST with Last-Event-ID returns 409 before creating a run.
The default POST response mode remains the legacy stream. The middleware forwards its frames without buffering or storing them. SDKs talking to an older streaming server read only the creation headers and close that response. Q1.3's restriction on unversioned historical/retained stream replay remains: separating creation does not make an otherwise unavailable replay safe. Permission-versioned historical replay remains Q5.3; poll authorised run status where streaming is unavailable.
SDK retry rules
Python and TypeScript default to two retries for GET/HEAD/OPTIONS, or explicitly keyed mutations with replayable bodies. They retry transport failures, 408, 429 and 5xx unless the response prohibits it. A 409 is retryable only with both X-Q-Retryable: true and X-Idempotency-Status: idempotency_in_progress. They do not automatically generate keys.
Unkeyed mutations, permanent conflicts, uncertain outcomes, received SSE responses and one-shot bodies are not retried. Response bodies are closed/cancelled before another attempt. Retry-After is capped at thirty seconds; cancellation interrupts backoff. Python task cancellation and TypeScript caller AbortSignal stop transport retries. Cancellation does not prove a server mutation was undone; retain the same key when investigating.
This changes earlier SDK behaviour: applications that depended on automatic unkeyed POST retries must supply and persist a key. Upgrading only the SDK cannot repair an older unsafe server's deduplication. Q2.3 qualifies installed-package retry/timeout transports; SDK support records the limits. Complete stream/replay semantics remain Q2.4.
Operator rollout
- Back up the database and upgrade through
20261007_160000_idempotency. Grant the non-superuser, non-BYPASSRLS application role USAGE onq_serviceand SELECT/INSERT/UPDATE onq_service.idempotency_operations. Do not disable forced RLS. Use durable PostgreSQL storage and retain these records across application restarts. - Drain old API instances and all old keyed in-flight operations before enabling traffic to the new middleware. Old Redis cache/lock entries and the legacy ingestion mapping do not contain request fingerprints. Reconcile known operations and expire the old retry window; do not silently import or discard their guarantees during mixed-version rollout.
- Deploy server and SDK changes together, update clients for the permanent conflict outcomes and receipt mode, and validate authentication/delegation/resource permissions. The GET stream restrictions above remain explicit.
- Qualify production concurrency, database pool/lease latency, unknown-state growth and API latency on representative workloads. The local component baseline does not measure this added authenticated database path.
- Schedule bounded cleanup of expired completed/non_replayable rows with an appropriately scoped maintenance role. A partial expiry index supports the scan. Never automatically delete unknown or in-progress records. Keep the cleanup and reconciliation audit outside private response/log payloads.
For example, per tenant, a maintenance transaction can select at most 1,000 terminal expired rows with FOR UPDATE SKIP LOCKED, then delete those selected keys. Bind q.tenant_id using set_config(..., true) in that transaction; use parameterised tenant values and require DELETE only on the maintenance role. Normal application traffic reclaims an expired terminal key atomically when it is reused, but does not sweep other records.
Uncertain-outcome reconciliation
Use X-Operation-ID to inspect the row through an operator connection bound to its tenant. The row exposes state, timestamps and an optional creation receipt. A failed/expired owner may still be executing: first stop or definitively account for the original execution. Compare the relevant domain resource, workflow/run history and any external provider receipt using the caller's retained request record. Do not log request bodies or credentials.
If the domain operation exists, record that finding in the operator audit and direct the caller to the currently authorised resource API. Keep the original key reserved. If its absence is conclusively established and the original worker can no longer execute, an operator may authorise a new operation/key under the domain's recovery procedure. Otherwise leave the outcome unknown and escalate for manual reconciliation. There is no automatic reset endpoint or timeout-based unknown cleanup.
The ownership record and external side effect are not one transaction. This prevents automatic duplicate dispatch within the contract but does not claim exactly-once external writes, durable delivery of every background task, or automatic recovery after a domain commit/receipt gap. Q1.6 handles index delivery; Q3.3 covers side-effect journalling. Do not downgrade/drop this table while replayable or unresolved operations remain. Prefer a forward fix that preserves ownership history.