# Vector Quantization and DiskANN: Fitting Indexes in Memory

*Published 2025-12-05 · Dmitry Shirokov · shirokoff.ca/blog/vector-quantization-diskann*

The PoC ran on a laptop: 200k chunks, HNSW, sub-10ms. Production was fifty times bigger, and the first honest sizing stopped the project for a week — the index alone wanted more RAM than the whole application tier had been budgeted. A 1536-dim float32 vector is ~6KB; ten million is ~60GB of raw vectors, plus another 20–40% for HNSW's neighbour lists. **Vector search isn't expensive because it's slow — it's expensive because the fast version lives in RAM.**

## The arithmetic, once

| Representation | Bytes/vector (1536-d) | 10M vectors | Recall impact |
|---|---|---|---|
| float32 | 6,144 | ~61 GB | Baseline |
| **float16** | 3,072 | ~31 GB | Negligible |
| **int8 scalar** | 1,536 | ~15 GB | Small (~1–2 pts) |
| **Product quantization** | ~96–192 | ~1–2 GB | Moderate; recovered by rescoring |
| **Binary (1 bit/dim)** | 192 | ~2 GB | Large alone; small with rescoring |
| Matryoshka → 512-d float32 | 2,048 | ~20 GB | Small, if trained for it |

Two decisions fall out: **float16 is nearly free** and plenty of deployments still store float32 for no reason; and the int8→PQ gap is an order of magnitude — "buy a bigger instance" vs "this fits comfortably."

## The three quantizations

**Scalar (int8)** — learn the per-dimension range, bucket into 256 levels. 4× smaller, faster distances (integer math, better cache behaviour), recall cost usually within noise for embedding models. Your default; if you run float32 and have never evaluated int8, that's the highest-return hour available.

**Product quantization** — split into *m* subvectors, k-means a codebook per subspace, store *m* bytes. 1536-d in 96 subvectors = 96 bytes, a 64× reduction, with distances from precomputed lookup tables. Real quantization error alone; with rescoring it recovers most of it, which is why PQ + rescoring is the workhorse for very large indexes.

**Binary** — keep the sign per dimension. 32× smaller, Hamming distance is a popcount, so you can scan enormous candidate sets. Too lossy alone; viable because binary is good enough to *rank roughly*, and a rough ranking is all a precise second pass needs. Modern variants improve on the naive sign trick — asymmetric comparison keeping the query at higher precision, and rotation-based schemes like RaBitQ.

## Rescoring is what makes compression safe

Search the compressed representation for candidates, recompute **exact** distances on those using full-precision vectors, return the reordered top-k. The knob is oversampling: to return 10, retrieve 100–200 candidates. Economics are excellent — reading a few hundred full-precision vectors from SSD costs milliseconds; keeping 10M in RAM costs 60GB permanently. Hence the standard shape: **compressed in memory, float32 on disk, rescore the top few hundred**, query full-precision throughout.

## DiskANN: built for SSD, not apologizing for it

Instead of compressing to fit RAM, build a graph that lives on SSD and is traversed in a small number of well-placed reads, with compressed vectors in memory guiding the search. The insight is access patterns, not compression: naive HNSW on disk is terrible because traversal is scattered random reads. DiskANN's construction gives better locality and batched reads, so a query costs a predictable handful of SSD accesses — RAM proportional to compressed vectors, storage to full vectors, latency higher than pure-RAM HNSW but fine for interactive use. It's why general-purpose databases (Postgres, document stores) grew DiskANN-style options: respectable vector search without a specialist engine's RAM budget.

## Matryoshka: compression before the index

Some models are trained so leading dimensions carry most of the information — truncate 1536 → 512 and keep most of the quality, a 3–6× win with no index machinery, composable with quantization. Requires a model trained for it; truncating an ordinary embedding degrades badly. Check the model card and measure on your own corpus.

## Three ways it goes wrong

**Wrong ground truth** — recall@k is defined against exact brute force on *your* data; comparing a quantized index to your previous approximate index hides compounding losses. Compute exact neighbours for a few thousand real queries once and keep them. **Filtering interaction** — a selective filter forces deeper exploration over degraded distances, so filtered recall can be far worse than the unfiltered number you tested. Measure **with filters on**. **Codebooks are trained** — PQ codebooks and scalar ranges are learned from a sample; a corpus distribution shift (new language, new document type, new embedding model) degrades recall slowly and silently. Re-train on material change, and monitor recall on a fixed query set.

## Choosing

<1M: float32/float16 HNSW — don't optimize a problem you don't have. 1–10M: **int8**. 10–100M: **PQ or binary + rescoring**, or **DiskANN-style**. >100M: DiskANN plus quantization, expect to shard. Memory-constrained, latency-relaxed: DiskANN. High QPS, small corpus: all in RAM, int8. Matryoshka available: truncate first.

And settle **what recall you actually need** before tuning — if a reranker reorders your top-50 anyway, 95% vs 98% recall@50 may be invisible in answer quality while costing a much larger index.

*Related: [Vector search: HNSW and IVF](https://shirokoff.ca/blog/vector-search-hnsw-ivf) · [Vector databases](https://shirokoff.ca/blog/vector-databases) · [pgvector on Postgres](https://shirokoff.ca/blog/pgvector-postgres-vector-database) · [RAG freshness](https://shirokoff.ca/blog/rag-freshness-incremental-indexing)*