Document ingestion quality and freshness

Q uses the same structural normaliser for uploads, Space files and connector indexing. The canonical public status operation is:

GET /v1/documents/{document_id}/ingestion
Authorization: Bearer YOUR_Q_API_KEY

It requires the read scope (including the documented AI scope compatibility rules), a tenant-bound credential and current document/Space access. The existing ingestion entry points supply the document ID. Status is available through authenticated HTTP and the prepared SDK helpers: Python q.knowledge.ingestion_status(document_id), TypeScript q.knowledge.ingestionStatus(documentId), and Go IngestionStatus. See SDK support for package publication and deployment compatibility. See authentication, capability discovery and index delivery.

A successful response includes:

{
  "document_id": "00000000-0000-0000-0000-000000000001",
  "document_version": 3,
  "state": "ready",
  "revision": 102,
  "indexed_revision": 102,
  "indexed_document_version": 3,
  "accepted_at": "2026-10-08T08:00:00Z",
  "updated_at": "2026-10-08T08:00:00.125Z",
  "last_indexed_at": "2026-10-08T08:00:00.125Z",
  "publication_lag_ms": 125,
  "error_code": null,
  "quality": {
    "parser_version": "q-structure-1",
    "format": "docx",
    "chunks": 4,
    "characters": 2400,
    "ocr_pages": 0,
    "structure": {"prose": 3, "table": 1}
  },
  "fresh": true
}

The timestamps/numbers above are illustrative. unknown identifies legacy documents with no ingestion receipt. queued and processing are unfinished; ready follows a committed index; failed reports a safe error code; deleted confirms tombstone publication. fresh=true means the indexed document version and revision match the latest request known to Q. It does not assert that Q has discovered every upstream edit. Poll with bounded exponential backoff; stop on failed and resolve its cause before submitting another ingestion request. A later request supersedes earlier work. Read status after ambiguous responses rather than inferring readiness from a workflow being accepted.

indexed_revision and last_indexed_at retain the last successful publication while a new attempt is pending or failed. The last good content can remain searchable, with fresh=false. A failed parse never publishes partial/empty replacement content. Explicit ACL changes apply to existing chunks during admission; current Space permissions also govern retrieval and status. A revoked reader receives no status data. Source paths, raw parser/provider errors, content and private ACLs are excluded from the response.

Retained structure and limitsCopy Anchor Link

The contract is q-structure-1. Input is capped at 16 MiB; extracted text at 2 million characters; serialised chunk text/metadata at 1 MiB; chunks at 4,096 with at most 4,000 characters each. Office archives allow at most 4,096 entries and 128 MiB expanded content; table rows/cells must fit the documented chunk budget. PDF limit is 1,000 pages. Binary/upload parsing uses a separate process with 90-second wall time, 60-second CPU time and a 1 GiB memory limit on Linux. These limits reject oversized input rather than silently truncating it.

OCR requires Tesseract and Poppler on the worker. Missing binaries return document_ocr_unavailable; empty OCR returns document_ocr_empty. Native OCR accuracy/languages, complex PDF layout and table inference require corpus qualification for the deployment. PDF tables retain extracted text layout; Q does not claim perfect semantic table reconstruction. Legacy .doc, .xls, presentations and arbitrary binary formats return document_format_unsupported. Other safe errors include document_parse_failed, document_parse_timeout, document_encrypted, document_size_limit, document_payload_limit, document_source_empty, document_dispatch_failed and document_index_failed.

Google Drive exports native documents/spreadsheets as DOCX/XLSX to preserve structure. Other connector implementations can return already extracted text or bounded content_bytes with a format_filename; binary input then uses the same parser as uploads. Incomplete connector implementations do not become ready by returning empty text.

Identity, updates and deletionCopy Anchor Link

Unchanged normalised chunks keep their IDs/vectors when unrelated sections are inserted. A changed chunk gets a new content-derived identity and the old row is tombstoned. Metadata-only changes update the document/chunk version without recomputing unchanged embeddings. Repeated identical chunks embed once within a publication but retain separate occurrence IDs. Identical content under different document IDs remains independently indexed and authorised; it is never silently skipped as a duplicate.

Chunk publication, document version, ready receipt and connector revision fence commit together. A newer accepted revision prevents older workers from publishing; repeated delivery recognises completed work. Deletion tombstones live chunks across all supported embedding dimensions and fences older deliveries. Identical refreshes do not invalidate retained evidence merely by updating last_seen_at.

The controlled local regression budget is five seconds for update/delete publication and permission revocation. The measurement starts at Q admission/processing and ends at an observed database/retrieval result. It is a test budget, not a hosted service SLO. publication_lag_ms measures admission to committed publication and includes Q queue time when admission was recorded at dispatch. Upstream polling, provider latency, broker recovery, production load and OCR accuracy require separate qualification.

Rollout and recoveryCopy Anchor Link

Apply 20261008_040000_ingestion_quality, drain/reconcile old ingestion histories, and deploy API and workers together. The migration removes the old unique tenant/content-hash index. Restoring that index on rollback may fail if independent same-content documents exist; do not delete documents to force rollback.

Keep the Q1.6 durable broker/revision configuration. Failed connector work remains subject to its retry/DLQ/replay policy. Receipt storage is not an upload dispatch outbox: a process failure between database admission and Temporal admission can leave a queued receipt. Reconcile it against Temporal before retrying. Required migrations, parser dependencies and worker isolation are deployment gates. No deployment or native OCR accuracy certification is implied by local test results.

Wait for a fresh indexed versionCopy Anchor Link

Use the SDK workflow helpers to wait for ingestion readiness with a deadline and an optional minimum document version.