RAG freshness: incremental indexing, deletes, and stale answers

The complaint was specific enough to be alarming: the assistant had quoted a returns policy that HR had withdrawn eleven weeks earlier. Not hallucinated — quoted, accurately, with a citation to a document that no longer existed on the intranet. The retrieval was working perfectly. It was retrieving from a snapshot of a world that had moved on.

This is the failure mode nobody designs for, because the demo indexes a corpus once and every subsequent conversation is about answer quality. But a production corpus is not a fixed set of documents; it's a stream of edits, supersessions and deletions, and a vector index that only knows how to add is a cache with no invalidation strategy. The most likely wrong answer your RAG system gives is not an invention. It's a correct quotation of something that used to be true.

So: how content actually changes, the four indexing patterns and when each is right, why deletes are harder than they look, when re-embedding is unavoidable, and how to measure staleness before a user does it for you.

Four ways a corpus changes, three of which break you

The middle two are where the pain is. An edited document is worse than it sounds: if the new version has fewer chunks than the old one, a naive "upsert by chunk id" leaves the extra old chunks orphaned in the index forever, still retrievable, still cited. That is the single most common freshness bug I find, and it produces answers that mix two versions of a policy in one paragraph.

The fourth row — access changes — is the one that turns a freshness bug into a security finding, which is why permission metadata belongs on the chunk and gets re-evaluated at query time rather than being baked in at index time.

The four indexing patterns

1. Full rebuild

Re-index everything on a schedule, swap the index atomically when it's done. Dismissed as naive, and it's the right answer more often than people admit — up to hundreds of thousands of chunks it's simple, self-healing (every bug in your incremental logic is erased nightly), and trivially correct on deletes because deleted documents simply aren't in the new build. Its costs are latency (freshness bounded by your rebuild interval) and embedding spend on unchanged content.

Use it when your corpus is bounded and a few hours of staleness is acceptable. Build to a new index and swap an alias — never mutate the live one, because a half-rebuilt index serving traffic is worse than a stale one.

2. Incremental by change feed

Subscribe to changes at the source and process only what moved. SharePoint and Google Drive expose change APIs, Confluence has webhooks, a database-backed corpus gives you CDC — the same log-based capture I've written about for Debezium and CDC, pointed at documents instead of rows. This is the right default for large or fast-moving corpora, and the freshness is as good as your queue latency.

Two requirements make it work. A stable document identity that survives renames and moves, because the file path is not an id. And a content hash per document, so a change event that didn't actually alter the text costs you nothing — SharePoint in particular will happily tell you a file changed because someone opened it.

3. Poll-and-diff

Walk the source, compare hashes against what you indexed, act on differences. Unglamorous, universally applicable, and correct on deletes by construction — anything in your index that isn't in the source listing gets removed. When the source has no usable change feed, this is the pattern, and it pairs well with a nightly cadence plus a faster feed for the sources that support one.

4. Query-time freshness check

After retrieval, verify the top-k chunks still exist and are still current, before generating. A cheap key lookup against your document table, filtering out anything retired. This isn't an indexing strategy — it's a safety net under whichever one you chose, and it's the control that would have prevented the incident I opened with. It costs a few milliseconds and it converts "confidently cites deleted policy" into "doesn't cite it."

graph TB
  SRC[("Document sources
SharePoint · Drive · DB · S3")] --> DETECT{"Change detection"}
  DETECT -->|"change feed / CDC"| Q["Queue: doc_id + op"]
  DETECT -->|"nightly walk"| DIFF["Poll + hash diff"]
  DIFF --> Q
  Q --> HASH{"content hash
actually changed?"}
  HASH -->|"no"| SKIP["Skip — free"]
  HASH -->|"yes"| PLAN{"operation"}
  PLAN -->|"upsert"| DEL1["Delete ALL chunks for doc_id"]
  DEL1 --> CHUNK["Re-chunk + embed"]
  CHUNK --> INS["Insert new chunks
version = n+1"]
  PLAN -->|"delete"| PURGE["Delete all chunks
+ tombstone in doc table"]
  INS --> IDX[("Vector index")]
  PURGE --> IDX
  IDX --> RET["Retrieve top-k"]
  RET --> CHECK{"Freshness check
doc still live + current?"}
  CHECK -->|"stale"| DROP["Drop chunk, backfill from k+1"]
  CHECK -->|"ok"| GEN["Generate with citations"]

Deletes are harder than they look

Three specific problems, each of which has bitten me.

You need document→chunk lineage. To delete a document you must know which vectors belong to it, which means storing doc_id as filterable metadata on every chunk and being able to delete by filter. Every serious vector store supports this; the mistake is generating chunk ids that don't carry the document, then discovering you can't clean up without a full scan.

Deletes are often asynchronous. Many vector indexes mark a vector deleted and reclaim the space during a later compaction or merge. Depending on the engine, a "deleted" vector may still be returned by a search until that happens. This is exactly the gap the query-time freshness check covers, and it's the reason I treat that check as mandatory rather than defensive.

Deletes are the operation nobody tests. Every ingestion pipeline I've reviewed has a test for indexing a new document. Almost none has a test that deletes one and then asserts it can no longer be retrieved. Write that test on day one — it is four lines and it is the difference between the incident that opened this article and a log line.

# The two operations that keep an index truthful. Note that update is
# delete-then-insert, not upsert — a document that shrinks from 12 chunks to 9
# leaves 3 orphans behind under upsert-by-chunk-id, and they stay retrievable.
def upsert_document(doc_id: str, text: str, meta: dict) -> None:
    digest = sha256(text)
    if store.get_doc_hash(doc_id) == digest:
        return                                   # unchanged: the cheap path

    chunks = chunk_structure_aware(text)
    vectors = embed([c.text for c in chunks])    # batch: it's the expensive call

    index.delete(filter={"doc_id": doc_id})      # ALL old chunks, unconditionally
    index.upsert([
        dict(id=f"{doc_id}::{i}", vector=v, metadata={
            "doc_id": doc_id, "chunk_seq": i,
            "acl": meta["acl"],                  # re-evaluated at query time
            "source_url": meta["url"], "indexed_at": now(),
        })
        for i, (c, v) in enumerate(zip(chunks, vectors))
    ])
    store.set_doc_hash(doc_id, digest)

def delete_document(doc_id: str) -> None:
    index.delete(filter={"doc_id": doc_id})
    store.tombstone(doc_id, at=now())            # tombstone, so the query-time
                                                 # check can catch stragglers that
                                                 # a lazy compaction still serves

When you have to re-embed everything

Three triggers, and only one of them is optional:

• The embedding model changes. Non-negotiable, and total: vectors from two models are not comparable, so a mixed index silently degrades retrieval in a way that's very hard to diagnose. Build a new index, run both in parallel long enough to compare recall on your eval set, then cut over.
• The chunking strategy changes. Also total, for the same reason — you're changing what a vector means.
• Content drifts far enough that the corpus isn't what it was. Rare, and the judgment call. Usually this shows up as a slow decline in retrieval recall on a fixed eval set, which is one more reason to have one.

Plan for re-embedding as a first-class operation with a versioned index name and an alias, not as an emergency. The teams that suffer here are the ones whose index name is hard-coded in five services, so a model upgrade becomes a coordinated deployment instead of an alias swap.

Measuring staleness before a user does

Three metrics, none expensive:

• Index lag — the p95 age of the newest indexed version versus the source's last-modified. This is your actual freshness SLO, and it should be on a dashboard with a threshold, not discovered during an incident.
• Orphan rate — chunks whose doc_id no longer exists in the source, sampled nightly. Should be zero; anything else means your delete path is broken and you now know before a customer does.
• Stale-citation rate — how often the query-time check drops a retrieved chunk. Small and non-zero is healthy (the net is doing its job); climbing means the indexing path is falling behind.

Add a handful of freshness cases to your eval set — questions whose correct answer changed when a document was updated. They're the only cases that fail specifically when the pipeline goes stale, and they'll catch the regression that a quality-focused suite sails past.

What to carry away

A RAG corpus is a stream, not a snapshot, and the likeliest wrong answer is a confident quotation of something retired. Handle all four kinds of change — add, edit, delete, and reclassify — and treat the last two as first-class rather than afterthoughts.

Pick the indexing pattern to fit the corpus: full rebuild with an atomic alias swap while you can get away with it, because it is self-healing and correct on deletes by construction; a change feed with stable document ids and a content-hash gate when volume or freshness demands it; poll-and-diff when the source offers nothing better. Under whichever you choose, put a query-time freshness check that drops retired chunks from the top-k — it is the cheapest safety net in the system and it prevents the failure that actually happens.

Delete by doc_id before inserting, never upsert by chunk id. Write the delete test on day one. Treat a change of embedding model or chunking as a full re-embed behind a versioned index and an alias. And watch index lag, orphan rate and stale-citation rate, so the first person to notice your index has fallen behind is you.