# RAG Freshness: Incremental Indexing, Deletes, and Stale Answers

*Published 2025-05-06 · Dmitry Shirokov · shirokoff.ca/blog/rag-freshness-incremental-indexing*

The assistant quoted a returns policy HR had withdrawn eleven weeks earlier. Not hallucinated — *quoted*, accurately, citing a document that no longer existed. Retrieval was working perfectly, against a snapshot of a world that had moved on. **A vector index that only knows how to add is a cache with no invalidation strategy**, and the likeliest wrong answer your RAG system gives is a correct quotation of something that used to be true.

## Four kinds of change

| Change | Index must | If it doesn't |
|---|---|---|
| New document | Chunk, embed, insert | Missing answers — visible, everyone handles it |
| **Edited** | Re-chunk, replace **all** its chunks atomically | Old and new chunks coexist; retriever picks the higher score |
| **Deleted/withdrawn** | Remove chunks immediately | Confident citation of retired content |
| **Access change** | Update filter metadata, or remove | Someone retrieves what they may no longer see |

Edits are worse than they sound: if the new version has fewer chunks, upsert-by-chunk-id orphans the extras forever, still retrievable, still cited — the most common freshness bug I find, and it mixes two policy versions in one paragraph. Access changes turn a freshness bug into a security finding, which is why permissions live on the chunk and are evaluated at query time.

## Four indexing patterns

**Full rebuild** — rebuild on a schedule, swap an alias atomically. Underrated: simple, **self-healing** (nightly erasure of your incremental bugs), trivially correct on deletes. Costs freshness latency and embedding spend on unchanged content. Never mutate the live index.

**Incremental by change feed** — SharePoint/Drive change APIs, Confluence webhooks, or CDC for a DB-backed corpus. Right default at volume. Needs two things: a **stable document identity** surviving renames (a path is not an id) and a **content hash**, because sources will tell you a file changed because someone opened it.

**Poll-and-diff** — walk the source, compare hashes, act on differences. Universally applicable and correct on deletes by construction: anything indexed but not listed gets removed.

**Query-time freshness check** — after retrieval, verify the top-k still exist and are current before generating. Not an indexing strategy; a safety net under whichever one you chose, and the control that prevents the opening incident. A few milliseconds.

## Deletes are harder than they look

**You need doc→chunk lineage** — `doc_id` as filterable metadata on every chunk, delete-by-filter. Chunk ids that don't carry the document mean you can't clean up without a full scan. **Deletes are often asynchronous** — many engines mark and reclaim during compaction, and a "deleted" vector can still be returned until then. **Deletes are the operation nobody tests** — every pipeline has an indexing test; almost none asserts a deleted document can no longer be retrieved. That test is four lines.

Update is **delete-then-insert**, not upsert: `index.delete(filter={"doc_id": doc_id})` before inserting the new chunks, gated by a content hash so unchanged documents cost nothing.

## When to re-embed everything

**Embedding model change** — non-negotiable and total; vectors from two models aren't comparable and a mixed index degrades retrieval in a way that's very hard to diagnose. **Chunking strategy change** — same, because you're changing what a vector means. **Content drift** — rare, shows up as slow recall decline on a fixed eval set. Plan re-embedding as a first-class operation behind a versioned index name and an alias; teams suffer when the index name is hard-coded in five services.

## Two bugs that cost me most

**Upsert-by-chunk-id on a shrinking document**: fourteen chunks edited down to nine, five orphans retrievable for months. It looks exactly like a hallucination and isn't — the citation points at a real document that no longer contains the text. **Trusting delete latency**: verified deletes returned success, a user retrieved one twenty minutes later because compaction hadn't run. The query-time check catches both.

## Measuring staleness

**Index lag** (p95 age of newest indexed version vs source last-modified) is your freshness SLO. **Orphan rate** (indexed `doc_id`s absent from the source, sampled nightly) should be zero. **Stale-citation rate** (how often the query-time check drops a chunk) — small and non-zero is healthy; climbing means indexing is falling behind. Add freshness cases to your eval set: questions whose correct answer changed when a document was updated.

## Carry-away

A corpus is a stream, not a snapshot. Full rebuild with an alias swap while you can; change feed with stable ids and a hash gate when you can't; poll-and-diff when the source offers nothing. Under all of them, a query-time freshness check. Delete by `doc_id` before inserting. Write the delete test on day one. Treat model or chunking changes as a full re-embed behind an alias. Watch index lag, orphan rate, and stale-citation rate.

*Related: [Debezium and CDC](https://shirokoff.ca/blog/debezium-cdc) · [Vector databases](https://shirokoff.ca/blog/vector-databases) · [Unity Catalog](https://shirokoff.ca/blog/unity-catalog) · [System design for data engineers](https://shirokoff.ca/blog/system-design-for-data-engineers)*