Recoverable connector index delivery

Connector sync publishes validated events to Redis Streams. A fixed set of worker slots awaits indexing before acknowledgement. Workers reclaim abandoned pending messages after a crash. PostgreSQL revision fences make duplicate and out-of-order delivery safe, including an older update that finishes after a newer delete.

A completed sync response means its events were accepted for delivery, not that indexing has completed. The result includes index_events_published and index_revision. Publication failure returns 502 connector_sync_or_index_delivery_failed and marks the sync failed; the API no longer hides queue outages. Q1.5's mutation-key/uncertain-outcome contract still applies to retries of the sync API.

ConfigurationCopy Anchor Link

The transport uses the shared Redis connection configuration (REDIS_URL, with the existing legacy Redis URL fallback). Verify the actual resolved endpoint. Provision durable persistence/replication, backups and no eviction for these keys; do not attach stream TTLs or run MAXLEN/XTRIM cleanup. The confirmation flag is an operator attestation, not an automatic persistence test. Configure Redis acknowledgement/fsync policy for the deployment's data-loss tolerance. The disposable local fixture intentionally has no persistence and only qualifies worker-process failure recovery.

Use Redis 6.2 or later for XAUTOCLAIM; the verified local fixture also exercises the newer deleted-pending-ID response. The existing Redis client uses a single server connection. Source/DLQ Lua transfers must execute on one server (or the same hash slot in a separately qualified cluster deployment). Multiple consumer groups are unsupported because successful source entries are deleted.

The old batch flush/max-events settings are not used to create background task buffers. Work remains broker-side until a slot is free. Scale per-process concurrency against database/provider capacity; fleet-wide fairness and production throughput remain separate qualification. A coroutine deadline cannot interrupt blocking native/CPU work, so the PostgreSQL publication fence remains authoritative even if a handler outlives its nominal deadline.

Event and version contractCopy Anchor Link

IndexEvent requires a non-empty tenant, canonical connector/document UUIDs, supported change type (create, update, delete), finite timestamp and positive 64-bit revision. It carries optional Space/connector/filename metadata and bounded remote identifiers. Its stable event_id derives from tenant, connector, document and revision; the fingerprint binds the semantic payload while excluding the observation timestamp. Invalid identity, unknown fields, absent revisions and conflicting equal revisions fail explicitly.

The sync API allocates a monotonic PostgreSQL revision before discovery, binds the connector implementation's _tenant_id to the authenticated tenant, and supplies the revision to every event. A batch may contain a document only once. Producer integrations must use allocate_sync_revision() before discovery and pass that revision to emit_events_from_sync_result(...); they must not use wall-clock time, arbitrary provider version strings, or a fresh revision for a transport retry. Retry the same immutable event payload/revision after an uncertain publication. Normal API clients do not supply revisions.

For updates, the document must already exist under the tenant. Fetch and embedding retain the document snapshot/version check. The final transaction commits document status, compatible permission-preserving chunks and the revision fence together. Deletes tombstone all three vector dimensions and update status/version in the same transaction. The fence also exists when the document row is missing. Older or exact duplicate events do nothing; a newer legitimate event may update the document again. A different connector claiming the same tenant/document or a different payload at the same revision is rejected.

These guarantees cover this connector event pipeline. Other ingestion/direct-storage paths retain their existing contracts. Current-source reads may observe changes made since discovery; upstream immutable snapshot validation, complete deletion feeds, pagination limits and full source-database outboxes remain connector-specific work.

Inspection and recoveryCopy Anchor Link

Use the operator's configured application environment, with trusted database/Redis credentials:

.venv/bin/python -m scripts.index_delivery
.venv/bin/python -m scripts.index_delivery --replay 1720000000000-0 --apply

Inspection reports source length, consumer-group pending/lag counts, DLQ length and up to twenty recent failure IDs/class codes/attempt counts. It omits private connector payloads and provider error text. /health/indexing exposes process-local success/failure/reclaim/DLQ counters and processing lag; it is not an end-to-end readiness or provider-capacity guarantee. Alert on growing pending age, delivery errors, DLQ growth, worker error codes and publisher failures.

For a valid failed event, inspect its authorised source and resolve the underlying failure first. Then replay that exact DLQ ID with --apply. The command validates the payload and atomically transfers it back to the source; a full source queue leaves the DLQ untouched. Repeated replay of an already transferred ID returns false. Index publication still checks its original revision, so replaying a pre-delete update cannot resurrect the document.

Malformed or legacy unversioned events cannot be automatically replayed. Quarantine them, establish authoritative source state and deletion history, and use a coordinated fresh sync with an allocated revision. Never guess a revision or blindly recreate documents. Failed events contain connector references and may include filenames, so restrict broker/DLQ access, backups and retention to the service/operator boundary. Failure logs and public counters contain class codes, not raw payloads.

A lost acknowledgement after a PostgreSQL commit is safe: the reclaimed event is recognised as already committed and acknowledged. Failure/cancellation before commit leaves the fence and data unchanged. A DLQ insertion error leaves the source pending. A message externally removed while pending is a durability violation and stops that consumer loop with an explicit error; the worker supervisor reports/retries the failure. Broker loss is not equivalent to ordinary worker recovery.

Rollout and rollbackCopy Anchor Link

1. Drain legacy publishers and workers. Inventory source, pending and DLQ entries and reconcile historical unversioned work against the source; retain deletion history. Do not run old and new acknowledgement schemes together.
2. Apply migration 20261007_180000_index_delivery. Grant the non-superuser, non-BYPASSRLS service role USAGE on q_service, SELECT/INSERT/UPDATE on q_service.index_document_versions and USAGE on q_service.index_sync_revision, alongside existing document/chunk grants. Keep forced RLS enabled.
3. Prepare the durable, non-evicting Redis deployment and set INDEX_REDIS_DURABILITY_CONFIRMED=true only after verification. Provision capacity/alerts and restrict direct stream writes. Keep source and DLQ key types correct.
4. Deploy producers and workers together, enable autostart in the intended worker processes, and verify accepted-event, pending, completion and DLQ paths in staging. Run representative latency/load and provider checks before production activation.
5. Preserve revision rows/tombstones and the sequence during backups, restores and rollback. Restore broker/source state coherently; revisions must not move backwards relative to retained/queued history. Prefer a forward fix. Downgrading the table/sequence destroys ordering guarantees and is unsafe while any event can replay.

No blanket DLQ purge, consumer deletion, stream trimming or automatic revision reset is provided. Index source/DLQ capacity is bounded by admission; revision tombstones require source-aware archival rather than a time-only deletion policy. User-visible per-document delivery status and broader connector capability qualification remain later public-platform work.

ADR 0008 · Sprint 21