# Building RAG on Azure: AI Search, Cosmos DB, pgvector, or Qdrant?

*Published 2026-01-22 · Dmitry Shirokov · shirokoff.ca/blog/rag-on-azure · as of early 2026*

The Azure RAG demoed beautifully and fell soft in production — confidently retrieving the wrong paragraph, missing the obvious one. The instinct was a bigger model. The problem was retrieval, and on Azure the retrieval decision that matters most is **which store you put your chunks in**. The model is a rented commodity; RAG quality is decided upstream of it — in chunking, the retriever, and the reranker. Companion to my [GCP](https://shirokoff.ca/blog/rag-on-gcp), [Snowflake](https://shirokoff.ca/blog/rag-on-snowflake), and [AWS](https://shirokoff.ca/blog/rag-bedrock-neptune) versions.

## The minimal pipeline

Five moving parts: a document source, a chunker, an embedding model (Azure OpenAI text-embedding-3), a retriever backed by a vector store, and a generation call (GPT-4o) — plus an evaluation harness built before you scale. You don't need a graph database, a feature store, and three vector engines to answer questions over documents.

**Fabric's role:** OneLake is the source-of-truth and ingestion layer (pipelines/notebooks land, clean, chunk). Fabric feeds RAG; in early 2026 it is not where the vector index should live. Clean split: Fabric-for-data, AI-Search-for-retrieval.

## Which retrieval store?

**Azure AI Search is the right default for document RAG**, because retrieval quality comes from hybrid search + semantic reranking, which it does natively, plus enterprise plumbing (security trimming, managed indexing). The others win when you want vectors next to data you already run.

| Store | Choose when | Catch |
|---|---|---|
| **Azure AI Search** | Default — best quality, least code | A service + index to run; tier sizing matters |
| **Cosmos DB for NoSQL** (DiskANN) | Operational data already in Cosmos — no ETL hop | Build hybrid + rerank yourself; watch RU cost |
| **pgvector** on Azure PostgreSQL | App + data already on Postgres; one datastore | Single-node scaling; assemble hybrid + rerank |
| **Qdrant** | Need dedicated-engine filtering/quantization at scale, will operate it | You run it (AKS/marketplace) |

- **Azure AI Search** — three GA capabilities decide quality: integrated vectorization (indexer chunks + embeds on the way in), hybrid search (BM25 + vector fused with RRF), and the semantic ranker (LLM re-scores top results — the single biggest lever). Plus native security trimming. Agentic retrieval existed but was **preview** in early 2026 — I shipped the GA classic pattern.
- **Cosmos DB for NoSQL** — native vector search with a DiskANN index; right when the grounding data already lives in Cosmos. Low latency, low RU cost, high-dimension support. It's a database, not a search engine: hybrid + reranking are yours; watch RUs.
- **pgvector on Azure PostgreSQL** — embeddings in the same DB as everything else; transactional, one thing to operate, SQL to combine distance with filters. Azure's DiskANN index for Postgres (GA 2025) softens the HNSW scaling ceiling. Single-node; you build hybrid + rerank.
- **Qdrant** — the pick when you genuinely need a purpose-built engine's filtering/quantization and will run it. Excellent, but on Azure the bar is "worth operating a system AI Search's managed service would handle" — high for most document RAG.

## What actually moved quality

- **Hybrid + semantic reranking, not pure vector** — the biggest, cheapest win. Vector-only misses exact terms (codes, proper nouns) and can't tell which top-20 match answers the question. Hybrid+rerank beat pure vector by a margin that dwarfed the embedding-model choice.
- **Chunking is a retrieval decision** — structure-aware chunks (split on headings/sections, not blind character counts), modest overlap, metadata on every chunk (source, section, date, access tags).
- **Security trimming from day one** — filter at the retriever with the user's identity, or RAG becomes a data-exfiltration tool. AI Search does it natively; on the databases, a filter predicate on every query. Retrofitting is painful.
- **Evaluate before you scale** — score retrieval recall (did the right chunk come back?) and groundedness (is the answer supported?). Azure AI Foundry has tooling; a spreadsheet of Q&A beats nothing. Recall tells you whether the problem is the store, the chunking, or the model — usually not the model.

```python
# Azure AI Search: hybrid + semantic rerank, integrated vectorization (query embedded server-side)
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
from azure.identity import DefaultAzureCredential

search = SearchClient("https://my-search.search.windows.net", "docs", DefaultAzureCredential())
results = search.search(
    search_text=question,
    vector_queries=[VectorizableTextQuery(text=question, k_nearest_neighbors=50, fields="content_vector")],
    query_type="semantic", semantic_configuration_name="default",
    filter="access_group eq 'enterprise'",   # security trimming at the retriever
    top=8,
)
```

## Lessons learned

Pure vector shipped to disappointment (fix: hybrid + semantic ranker — should've been the start). Over-chunking retrieved context-free fragments (bigger structure-aware chunks fixed answers overnight). Reaching for a graph DB we didn't need over a few-thousand-doc corpus (a plain vector index beat it). Chasing preview agentic retrieval instead of the GA pattern. Skipping evaluation — the project that "felt fine" had a recall problem a customer found first.

## Carry-away

The model is the commodity; retrieval is the product. Default to Azure AI Search (hybrid + semantic ranker + security trimming, integrated vectorization removes the ingestion job). Move to Cosmos/pgvector to keep vectors next to data you run, Qdrant for a dedicated engine you'll operate. Let Fabric be the data/ingestion layer; don't force the index into it. Then: ship hybrid+rerank not pure vector, treat chunking as retrieval, trim by security from day one, and measure recall + groundedness before you scale.

*Related: [Fine-tuning vs RAG vs Prompting](https://shirokoff.ca/blog/fine-tuning-vs-rag-vs-prompting) · [Vector Databases radar](https://shirokoff.ca/architecture-radar/vector-databases) · [Fabric vs Databricks](https://shirokoff.ca/blog/fabric-vs-databricks) · [RAG on GCP](https://shirokoff.ca/blog/rag-on-gcp) · [RAG on Snowflake](https://shirokoff.ca/blog/rag-on-snowflake)*