Embedding compatibility and re-indexing

Q1.4 binds knowledge ingestion and retrieval to the same model configuration. This change requires a data migration before deploying over legacy knowledge vectors. No database DDL is added, and no production migration is performed by the local verification suite.

Contract and selectionCopy Anchor Link

Each chunk's server-owned metadata._embedding records:

{
  "model": "text-embedding-3-small",
  "version": "deployment-2",
  "dimensions": 1536,
  "purpose": "knowledge",
  "mode": "provider",
  "task_profile": "retrieval-v1"
}

Knowledge ingestion, connector events, RAG search and the search API resolve the knowledge purpose. Selection uses that tenant's purpose pin, then its default pin, then the configured tenant/system preference. A pin carries model, dimensions and version. Existing pins without a version use the registered application version (1.0.0); this does not prove historical vector provenance. Configure pins through the existing operator service set_tenant_embedding_pin(tenant_id, model_id, purpose="knowledge", dimensions=1536, version="deployment-2") in the intended application environment. Pin changes do not migrate documents automatically.

Explicit model overrides must name a registered profile. Matching the pin's canonical model and dimensions retains the pinned version; a different override selects a different contract and can require re-indexing before search. Canonical aliases remove the OpenAI prefix for supported v3 models and map bare gemini-embedding-001 to gemini/gemini-embedding-001. Vertex routes remain distinct. A dimension suffix such as :1536 must agree with any separate dimensions argument. Supported storage dimensions are 768, 1,536 and 3,072, subject to the selected model's limits.

Q passes dimensions for OpenAI v3 models and output_dimensionality for supported reduced Gemini vectors. Knowledge queries and passages use the provider's corresponding retrieval tasks. See the OpenAI embedding API and Gemini embedding documentation. Provider access remains through LiteLLM. Local tests use synthetic responses and do not establish live-provider conformance or quality.

EMBEDDINGS_OFFLINE_MODE selects deterministic synthetic vectors without reading tenant pins. Their offline-synthetic-v1 provenance and cache keys differ from provider vectors. They cannot be substituted into a provider index. Application version labels must change when an operator changes a provider deployment or embedding semantics; Q cannot detect an upstream model change hidden behind the same identifier.

API and error behaviourCopy Anchor Link

The native embedding request accepts purpose (default default), input_type (document or query, default document) and dimensions. Its response includes resolved model_id, model_version, purpose, dimensions and vectors; X-Q-Embedding-Contract identifies the contract. To generate vectors intended for knowledge retrieval, explicitly use purpose="knowledge" and the appropriate input role.

The OpenAI-shaped router also accepts these fields, returns the actual resolved model and supplies X-Q-Embedding-Version and X-Q-Embedding-Contract. Existing routing limitation: app/main.py registers the native router before the OpenAI-shaped router at /v1/embeddings; the native request shape currently wins. /api/v1/embeddings also uses the native shape. Q1.4 updates both implementations without changing that pre-existing route precedence. Route/schema consolidation and generated SDK migration belong to Q2.1; do not assume an OpenAI client can use the colliding route yet.

Compatibility checks use current Q1.3 permissions and the same document/date/type/metadata filters as ranking. An inaccessible incompatible document must not block another user's compatible collection. Equal dimensions alone never establish compatibility. Mixed authorised collections fail before returning evidence; a document filter can isolate a fully migrated document. Keyword search remains available during migration. ENABLE_BLENDED_SEARCH no longer enables dimension-only fallback on the search route.

Operator migrationCopy Anchor Link

Use a reviewed application environment with access only to the intended tenant's data and provider configuration. The command does not ask interactively and is read-only unless --apply is supplied. Examples use placeholders, not real tenant or document identifiers.

1. Back up the tenant configuration and documents. Stop or coordinate ingestion and pin changes for the migration window. Inventory all live vectors:

   Copy Text

   .venv/bin/python -m scripts.reindex_embeddings --tenant TENANT_ID

2. Choose and save the target knowledge pin, including a deployment revision. Keep keyword retrieval available while the vector collection is mixed. Preview each selected document:

   Copy Text

   .venv/bin/python -m scripts.reindex_embeddings --tenant TENANT_ID --document DOCUMENT_UUID

3. After reviewing the preview, explicitly regenerate selected documents:

   Copy Text

   .venv/bin/python -m scripts.reindex_embeddings --tenant TENANT_ID --document DOCUMENT_UUID --apply

   Repeat --document to select several documents. Each document is a separate transaction; a failure stops subsequent work and does not undo earlier successful documents. The command reads stored text, preserves permission labels, rejects conflicting copies and never infers the old model from dimension. If those stored copies are not authoritative, use the normal source-ingestion flow instead.

4. Repeat inventory and verify searches with the intended authenticated users/Spaces. Compare retrieval quality on a representative holdout before enabling vector traffic. The full candidate set must have the selected contract. Existing unchanged text is re-embedded whenever provenance differs; repeating a completed re-index with the same contract reuses compatible stored vectors.

Provider or vector-validation failure leaves the current document's old live rows intact. Successful publication tombstones other dimension representations. A changed versioned ingestion is rejected rather than overwritten. All writers and pin changes should still be coordinated: this command does not provide a tenant-wide transaction or a general lock over external producers.

Rollback requires selecting the former pin and regenerating every changed document under that contract, or restoring a reviewed database backup. Merely reverting the pin leaves changed rows incompatible. Same-dimension re-indexing replaces active vectors in place; tombstones are not a complete rollback archive. Never remove _embedding to bypass compatibility checks.

Legacy split generation/save ingestion activities now require tenant context and a resolved contract; drain or upgrade old queued payloads. The maintained ingestion workflow uses the atomic smart-ingestion activity. The legacy 3,072-dimensional backfill now requires a tenant knowledge pin at 3,072 dimensions and uses the same re-index service.

Boundaries and verificationCopy Anchor Link

This contract covers the three local knowledge-document tables. Separate semantic/episodic memory, private codebase, skills, taxonomy and remote-provider indices need their own migration; do not label them as covered. Embedding caches store vectors, not retrieval permissions, and include tenant, contract, purpose and query/document role. Q1.3's restrictions on retained evidence and answer caches remain.

Run local checks through scripts.dev or scripts.release.check so synthetic settings replace ambient credentials. Live provider conformance, full hosted CI, generated clients and representative compatibility-query latency remain release qualification. ADR 0006 records the decision; Sprint 19 records acceptance criteria.