Diagnose Q requests and durable runs
Follow correlation IDs and owner-authorised diagnostics across a failed or slow run.
Retain X-Request-ID from every response and the agent_id/run_id from an accepted durable run. A supplied request ID is accepted only when it contains 1–128 ASCII letters, digits, underscores, full stops, colons or hyphens and begins with a letter/digit. Otherwise Q replaces it. IDs are correlation labels, never authentication. Do not put personal data in them.
Diagnose an owned run
curl --fail-with-body "$Q_API_BASE_URL/agents/$AGENT_ID/runs/$RUN_ID/diagnostics" \
-H "Authorization: Bearer $Q_API_KEY"This typed endpoint requires ai:answer and the exact credential principal that owns the version-2 run. It returns private, non-cacheable metadata without prompts, source contents, tool arguments/results, raw errors, provider references, webhook URLs or secrets. Another tenant or principal receives 404. Keep using the original principal after a retry; a replacement API key is a different principal. The diagnostic window is limited by the tenant's current trace_days; older/disabled diagnostics also return 404. A database outage is 503, not an empty or successful result.
| Evidence | Interpretation |
|---|---|
request_id, run_id, timestamps | Follow the original API request through persisted dispatch and worker execution. The response header identifies the diagnostic GET; the body identifies the original run request. Older runs may have no original ID. |
reason, failure_code, next_action | Controlled guidance for dispatch/worker waiting, active execution, input/approval, failure, deadline reconciliation, cancellation or unresolved effects. Known safe failure codes identify policy/capacity/provider/budget failures; unrecognised or free-text errors remain null. |
elapsed_ms, queue_ms | Time from acceptance to completion/current time, and acceptance to first worker start. These are wall-clock values, not sums of overlapping stages. |
latest_attempt, stages_recorded_at, stages | Bounded timings from the latest finished worker attempt. Missing timings mean unobserved; a running attempt or killed worker may not have flushed them. Retries and parallel stages are not additive. Provider timings cover dispatch/response setup; completed provider records describe usage separately. |
provider | Observed attempts/tokens/cost and incomplete observations. Missing reported cost stays null. It is not a billing statement. |
billing | Separate held, provisionally priced and confirmed spend for this run. No attempts means no ledger evidence, not proof that the provider charged nothing. |
tool_states, delivery_states | Pending/unknown effects and webhook deliveries. Run completion does not imply receiver completion. No delivery rows do not prove delivery or a subscription. |
replay_available, last_sequence | Whether the retained journal is within its expiry. Actual replay still requires current evidence permissions and may reject a cursor or revoked result. |
Follow the existing lifecycle, event replay, tool, spending and completion contracts to recover. Before starting replacement work, reconcile any accepted run, reservation or external effect; a timeout does not prove that nothing happened.
Timing metadata is retained with the execution ledger. trace_days controls diagnostic visibility and disables new timing capture when zero; it does not physically erase required execution, billing or effect records. Apply the separate data governance retention and erasure procedures.
Service objectives and status
GET /v1/devtools/service-objectives (when ENABLE_DEVTOOLS is enabled) returns counts for the authenticated principal's retained version-2 runs, with candidate completion and start-delay objectives. Completion excludes cancelled and still-active runs. Queue timing measures first start; pending work older than 30 seconds is bad, and a failure/expiry without a start is bad after 30 seconds. The creation cohort is at most 30 days, shortened by governance retention. Zero eligible observations returns null ratios and null budgets. Targets are candidates, not a production SLA; in-progress counts must be considered alongside the completion ratio.
GET /v1/status probes this API instance's schema and required Redis/Temporal dependencies. scope=instance_dependencies is not fleet/provider availability. Provider-backed capabilities remain listed as unverified until external operational evidence exists; a configured key/model is insufficient. latency_ms contains nulls because process-local samples cannot describe fleet latency. Capability discovery describes supported/configured operations, while status describes current dependency readiness.
DevTools request traces now return safe measurements for the authenticated owner, without historical source IDs, plans or trace bodies. Historical traces without trusted owner attribution are not exposed. Store failure is explicit. Use authorised content/resource APIs when source material is needed.
For escalation, provide the request/run IDs, diagnostic timestamps/reason and response status, without prompts or credentials. Operators can hash IDs with SHA-256 to match structured logs and critical-path spans. See the service objectives runbook for metrics, error budgets and incident response.