The proof of concept ran on a laptop. Two hundred thousand chunks, HNSW, sub-ten-millisecond queries, everyone delighted. The production corpus was fifty times larger, and the first honest sizing exercise produced a number that stopped the project for a week: the index alone wanted more RAM than the entire application tier had been budgeted.
That arithmetic is worth doing before you commit to an architecture, because it is unforgiving and completely predictable. A float32 vector of 1536 dimensions is about 6KB. Ten million of them is roughly 60GB of raw vectors, and a graph index like HNSW adds its neighbour lists on top — typically another 20–40% depending on how you tune the graph's connectivity. That is your memory bill before a single byte of payload, metadata or operating system.
Vector search is not expensive because it's slow; it's expensive because the fast version lives in RAM. There are exactly two ways out of that — make each vector smaller (quantization) or stop keeping them all in memory (DiskANN and friends) — and a third technique, rescoring, that is what makes the first two safe to use. This is how they work and how to pick.
The memory arithmetic, once
| Representation | Bytes per 1536-dim vector | 10M vectors | Typical recall impact |
|---|---|---|---|
| float32 (raw) | 6,144 | ~61 GB | Baseline |
| float16 / bfloat16 | 3,072 | ~31 GB | Negligible |
| int8 scalar quantization | 1,536 | ~15 GB | Small — often ~1–2 points |
| Product quantization (typical config) | ~96–192 | ~1–2 GB | Moderate; recovered by rescoring |
| Binary (1 bit/dim) | 192 | ~2 GB | Large alone; small with rescoring |
| Matryoshka truncation to 512 dims, float32 | 2,048 | ~20 GB | Small, if the model was trained for it |
Two observations from that table that change decisions. First, float16 is nearly free — half the memory for a recall difference most systems can't measure — and a surprising number of deployments are still storing float32 for no reason. Second, the gap between int8 and product quantization is an order of magnitude, which is the difference between "buy a bigger instance" and "this fits comfortably."
The three quantizations
Scalar quantization: the free lunch
Map each float32 dimension onto an int8 by learning the value range per dimension and bucketing into 256 levels. Four times smaller, distance computations get faster because integer arithmetic is cheaper and more of your data fits in cache, and the recall cost is usually within noise for embedding models — whose components are well-behaved and roughly normally distributed.
This should be your default. If you are running float32 vectors in production and have never evaluated int8, that's the highest-return hour available to you.
Product quantization: the big compression
Split the vector into m subvectors, run k-means on each subspace to learn a codebook of (typically) 256 centroids, and store the vector as m bytes — one centroid id per subspace. A 1536-dim vector with 96 subvectors becomes 96 bytes: a 64× reduction.
Distances are computed approximately using precomputed lookup tables, which is fast. The cost is real quantization error, because you've replaced each subvector with its nearest centroid. Alone, PQ loses meaningful recall. With rescoring — below — it recovers most of it, which is why PQ plus rescoring is the workhorse configuration for very large indexes.
Binary quantization: one bit per dimension
Keep only the sign of each dimension. 32× smaller than float32, and Hamming distance is a popcount, so comparisons are extraordinarily fast — you can scan enormous candidate sets cheaply.
It sounds too lossy to work, and used alone it usually is. What makes it viable is the shape of the problem: binary is good enough to rank roughly, and you only need a good rough ranking to produce candidates that a precise second pass can reorder. Modern implementations improve the naive sign trick considerably — asymmetric comparisons that keep the query at higher precision, and rotation-based schemes such as RaBitQ that give better guarantees than plain sign-bit hashing. This is the technique behind the "binary quantization with rescoring" configurations that have shown up across vector engines and search products over the past couple of years.
graph LR Q["Query vector
float32, kept precise"] --> COARSE["Coarse search
over compressed vectors
(PQ or binary, in RAM)"] COARSE -->|"top 200-1000 candidates"| RESCORE{"Rescore"} FULL[("Full-precision vectors
SSD or a separate store")] --> RESCORE RESCORE -->|"exact distances on
a few hundred vectors"| TOPK["Final top-k
recall close to uncompressed"] NOTE["Compressed set is small enough for RAM;
full-precision set is read rarely and sequentially"] -.-> FULL
Rescoring is the idea that makes aggressive compression safe, and it's the one most teams miss. The compressed index only has to get the candidate set right, not the final order — so you can afford a lossy representation for the scan, then pay for exact distances on a few hundred vectors. Note the asymmetry: the query stays full precision throughout, which costs nothing and helps measurably.
Rescoring, and the oversampling knob
Search on the compressed representation to get candidates, then recompute exact distances on those candidates using full-precision vectors, and return the reordered top-k. The one parameter that matters is the oversampling factor: to return 10 results you might retrieve 100–200 candidates from the compressed index.
The economics are excellent. Reading a few hundred full-precision vectors from SSD or a secondary store costs a handful of milliseconds and a trivial amount of I/O; keeping ten million of them in RAM costs 60GB permanently. That trade is why "binary or PQ in memory, float32 on disk, rescore the top few hundred" has become the standard shape for large indexes.
Tune oversampling empirically against your own recall target — it's the knob with the clearest recall-versus-latency curve in the whole system, and it's usually the first thing I'd adjust when a quantized index underperforms.
DiskANN: designing for SSD instead of apologizing for it
The other direction. Rather than compressing vectors to fit RAM, DiskANN builds a graph index that lives on SSD and is traversed with a small number of well-placed reads, keeping a compressed copy of the vectors in memory to guide the search.
The key insight is about access patterns rather than compression. A naive HNSW index on disk performs terribly because graph traversal produces scattered random reads, and each hop waits on I/O. DiskANN's construction produces a graph with better locality and a search procedure that batches reads, so a query costs a small, predictable number of SSD accesses. In practice: RAM proportional to the compressed vectors (small), storage proportional to the full vectors plus graph (cheap), latency higher than pure-RAM HNSW but low enough for interactive use — single-digit to low-tens of milliseconds in well-tuned deployments.
It has become the standard answer for large managed vector search, which is why you now find DiskANN-derived indexes inside cloud databases as well as dedicated engines — Postgres and document databases both grew DiskANN-style options precisely because it lets a general-purpose database offer respectable vector search without demanding a RAM budget shaped like a specialist engine's. If you're weighing engines rather than index types, that's the vector database comparison.
Matryoshka embeddings: compression before the index
Worth knowing because it's the cheapest option of all when available. Some embedding models are trained so that the leading dimensions carry most of the information — truncate a 1536-dim vector to 512 or 256 and you keep most of the quality. That's a 3–6× reduction with no index-side machinery at all, and it composes with quantization: truncate, then int8, and you're at a small fraction of the original footprint.
The requirement is that the model was trained for it. Truncating an ordinary embedding degrades badly, because there's no reason for the early dimensions to be more important than the rest. Check the model card, and measure on your own data rather than trusting the general claim — the retained-quality numbers are corpus-dependent.
Three ways compression goes wrong in production. Measuring recall against the wrong ground truth. Recall@k is defined against exact brute-force search on your data. Teams routinely compare a quantized index to their previous approximate index and conclude compression was free, when both had already lost the same 4% and the losses compounded. Compute exact neighbours for a sample of a few thousand real queries once, keep it, and measure everything against that. Compression that interacts with filtering. A selective metadata filter forces the index to explore much further to find k matching candidates, and on a compressed index the extra exploration is over degraded distances — so filtered recall can be far worse than the unfiltered number you tested. If your queries filter by tenant or date, measure recall with the filters on. Forgetting that quantization parameters are trained. PQ codebooks and scalar ranges are learned from a sample of your vectors. Feed in a corpus whose distribution shifts substantially — a new language, a new document type, a different embedding model — and the codebook no longer represents the data. Recall degrades slowly and silently, which is the worst failure shape. Re-train the codebook when the corpus changes materially, and put a recall check on a fixed query set in your monitoring so the drift is visible.
Choosing, in practice
| Situation | Start with |
|---|---|
| < 1M vectors | float32 or float16 HNSW. Don't optimize a problem you don't have. |
| 1M – 10M vectors | int8 scalar quantization. Near-free, and usually sufficient. |
| 10M – 100M vectors | PQ or binary + rescoring, or a DiskANN-style index if your engine offers one. |
| > 100M vectors | DiskANN-style plus quantization, and expect to shard. |
| Memory-constrained but latency-relaxed | DiskANN. Trade milliseconds for gigabytes; it's usually the better trade. |
| Very high query rate, small corpus | Everything in RAM, int8. Compression buys nothing you need. |
| Model supports Matryoshka | Truncate first, then quantize. Cheapest win available. |
And two process notes that matter more than the choice itself. Measure recall on your own data — published benchmarks use corpora whose embedding distributions differ from yours, and the compression that loses 1% on one dataset can lose 8% on another. Decide what recall you actually need, which for RAG is usually lower than instinct suggests: if a reranker reorders your top-50 anyway, the difference between 95% and 98% recall@50 may be invisible in answer quality while costing you a much larger index. That question — how much retrieval quality your pipeline actually converts into answer quality — is the one worth settling before you spend a week tuning.
What to carry away
Do the memory arithmetic before you choose an architecture: raw float32 vectors plus a graph index is tens of gigabytes at ten million chunks, and that number decides more designs than latency does. Then work down the ladder. float16 is nearly free; int8 scalar quantization should be your default; product and binary quantization buy an order of magnitude more and need rescoring to be safe — search compressed, then recompute exact distances on a few hundred candidates, keeping the query at full precision throughout.
When memory is the binding constraint rather than latency, DiskANN-style indexes are the better answer than more compression: they're built for SSD access patterns instead of tolerating them, which is why they now show up inside general-purpose databases. And if your embedding model supports Matryoshka truncation, take that first — it composes with everything else.
Measure recall against exact brute-force ground truth on your own data, with your filters applied, and re-train quantization codebooks when the corpus shifts, because that failure is silent. Most of all, settle what recall your pipeline actually needs before optimizing — a reranker downstream may be quietly absorbing the difference you're about to spend a week and a large instance chasing.