Semantic Cache Is Not Semantic Memory: Wiring Both Into LangGraph, Semantic Kernel, and Azure/AWS/GCP

Two things happened on the same project, three weeks apart, and it took me longer than I'd like to admit to see they were the same root cause. First: a team lead told me proudly that they'd "added semantic memory" to their support agent by wiring in a Redis-backed semantic cache in front of the LLM call. Great idea, wrong tool — three days later a user complained the agent still didn't remember their account tier from a conversation two days earlier, because a cache that short-circuits repeat questions has nothing to do with an agent recalling facts across sessions. Second: during a multi-cloud migration, someone ported a semantic-cache config from Bedrock straight into Azure API Management, expecting the same behavior, and got confidently wrong answers served from cache for genuinely different questions — because what they'd been using on Bedrock wasn't semantic caching at all. It was prefix caching, an entirely different mechanism with entirely different guarantees, and nobody had noticed the difference until it started lying.

Both mistakes trace back to the same habit: treating "cache," "memory," and "context" as roughly interchangeable words for "the agent knows stuff." They're not. This piece is about the two distinctions that actually matter — cache versus memory, and prefix caching versus semantic caching — and how to wire the right thing into LangGraph, CrewAI, LlamaIndex, Semantic Kernel, and each of the three big clouds without repeating either mistake. If you want the deeper taxonomy of memory types and the security story around memory poisoning, I already covered that ground in AI Agent Memory: The Infrastructure Layer Nobody Told You About — this article assumes you've read that or don't need it re-explained, and focuses entirely on the integration and caching side that piece left out.

What problem is agent memory actually solving?

An LLM API call is stateless — every request carries its own complete context, and nothing persists on the model side between calls. Agent memory exists to work around that in two different time horizons. Short-term memory is what makes a single conversation coherent: the running exchange, tool outputs, and intermediate reasoning for the current session, sometimes called working memory or in-context memory because it lives in the prompt itself and disappears when the session ends. Long-term memory is what persists across sessions, and it splits into three well-established types: episodic memory (what happened — specific past interactions and events), semantic memory (accumulated facts and world knowledge — not to be confused with semantic caching, which is a different thing entirely and the whole point of the next section), and procedural memory (learned patterns for how to do things — usually a system prompt or fine-tune rather than something genuinely learned from experience).

That taxonomy answers "what does the agent know and for how long." It says nothing about cost or latency, which is a separate problem: sending the same or similar prompts through an LLM repeatedly is expensive and slow, independent of whether the agent needs to remember anything at all. That's the gap caching fills — and it's where the confusion starts, because "semantic" shows up in both "semantic memory" and "semantic caching," describing two mechanisms that share a similarity-search technique but solve unrelated problems.

Why isn't semantic caching just another kind of memory?

A semantic cache stores pairs of {prompt embedding → response} and, on a new query, checks whether an existing entry is similar enough — above a configured similarity threshold — to return without calling the LLM at all. Its entire job is to avoid a redundant model call for a question that's already been answered in essentially the same way before. It has no concept of "this user," no notion of accumulating facts over time, and no mechanism for tracking what happened when. A semantic cache that's never seen a particular user before will happily return a cached answer to their first-ever question, if someone else asked something similar enough last week.

Memory does the opposite job at the opposite point in the pipeline. It doesn't decide whether to call the LLM — it decides what goes into the call, by retrieving relevant facts and injecting them into the prompt before the model ever runs. Cache sits upstream of the decision to call the model; memory sits inside the call itself. Conflating them is exactly how my team lead ended up disappointed — a cache hit on a semantically similar but user-specific question would, at best, do nothing for personalization, and at worst return a different user's cached response entirely if the cache wasn't scoped correctly (more on that failure mode later).

flowchart LR
    Q(["Incoming query"]) --> C{"Semantic cache lookup
embedding similarity"}
    C -->|"hit ≥ threshold"| R1["Return cached response
— LLM never called"]
    C -->|"miss"| MEM["Memory retrieval
episodic + semantic + procedural"]
    MEM --> PROMPT["Assembled prompt
system + retrieved memories + query"]
    PROMPT --> LLM["LLM call"]
    LLM --> R2["Response"]
    R2 -.->|"async"| CSTORE["Cache store
prompt embedding → response"]
    R2 -.->|"async"| MSTORE["Memory write
extract & store facts"]
          

What's the difference between prefix caching and semantic caching?

This is the distinction that actually caused the multi-cloud incident, and it's the one I see conflated most often because every cloud provider calls its version of caching some variant of "prompt caching," regardless of which of two genuinely different mechanisms it is. Prefix caching — what Amazon Bedrock calls prompt caching, what Google Vertex AI calls context caching, and what Anthropic's own API calls prompt caching — caches a literal, identical, repeated prefix of tokens: your system prompt, tool definitions, few-shot examples, or a large shared document that appears at the start of many requests. It matches exactly, byte for byte, up to the cache breakpoint. There's no similarity search and no threshold — it either matches or it doesn't, which means it's completely safe. A prefix-cache hit can never return the wrong answer, because the cached content is the input tokens, not the output.

Semantic caching — what GPTCache, Redis's RedisSemanticCache, and Azure API Management's azure-openai-semantic-cache policies do — caches full prompt-to-response pairs and matches on embedding similarity across genuinely different prompt text. "What's your refund policy?" and "how do refunds work?" can hit the same cache entry even though the token sequences share almost nothing. That's the entire value proposition — catching paraphrases a prefix cache would never match — and it's also the entire risk. Get the similarity threshold wrong and the cache will confidently return an answer to a question that wasn't quite the one being asked.

How do LangGraph, CrewAI, and LlamaIndex actually wire memory?

Every major framework has converged on the same shape — a short-term store scoped to the current run and a long-term store scoped across runs — but the defaults and production-readiness vary enough to matter.

LangGraph: checkpointer plus store, not either/or

LangGraph splits persistence into two separate primitives on purpose. The checkpointer persists a thread's graph state — conversation continuity, tool results, human-in-the-loop pauses, and fault recovery — and is thread-scoped, short-term memory. The store persists application data outside the graph state entirely, organized as namespaced documents, and is meant for long-term, cross-thread memory: user preferences, accumulated facts, anything that should survive a returning user opening a brand-new conversation. The single most common architecture mistake I see is compiling a graph with only one of the two — usually just the checkpointer, because it's the one the quickstart tutorial shows first.

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.postgres import PostgresStore

checkpointer = PostgresSaver.from_conn_string(DB_URI)   # short-term: thread-scoped state
store = PostgresStore.from_conn_string(DB_URI)          # long-term: cross-thread memory

graph = builder.compile(checkpointer=checkpointer, store=store)

# writing a long-term memory, namespaced per user
store.put(("memories", user_id), key="pref-language", value={"language": "Python"})

# reading it back in a later, unrelated thread
hits = store.search(("memories", user_id), query="preferred programming language")

MemorySaver is development-only — it resets on every process restart, which is fine for iterating locally and a production incident waiting to happen anywhere else. PostgresSaver is the production default: horizontally scalable, crash-recoverable, and it's the same database most teams already run for everything else.

CrewAI: memory as a flag, with one production trap

CrewAI's memory system is architecturally three pieces: short-term memory (ChromaDB-backed, resets every kickoff() call, exists only for the current crew run), long-term memory (SQLite-backed, persists lessons and outcomes across runs), and entity memory (a RAG-based store specifically for facts about people, companies, and other named entities the crew encounters). Turning it on is a single flag.

from crewai import Crew

crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, write_task],
    memory=True,
    embedder={"provider": "openai", "config": {"model": "text-embedding-3-small"}},
)

Here's the trap: the default long-term and entity memory backends write to local file storage. That's invisible in development and a silent data-loss bug the moment you deploy to a container or serverless function that doesn't persist a local filesystem across invocations — the crew looks like it's "forgetting" for no obvious reason, because every cold start is a fresh, empty SQLite file. Point long_term_memory and entity_memory at a durable backend (Postgres, S3-backed storage, or a hosted service like Mem0) before this reaches production, not after someone notices.

LlamaIndex and Semantic Kernel: the vector-store abstraction pattern

LlamaIndex's ChatMemoryBuffer handles short-term conversational context, and its composable memory modules pair that with a VectorMemory for retrieval-augmented long-term recall — the same short-term/long-term split as everyone else, expressed as composable objects. Semantic Kernel takes a more deliberately pluggable approach: a single VectorStore abstraction with connectors for Azure AI Search, Qdrant, Redis, Pinecone, Chroma, Weaviate, Milvus, and an in-memory implementation for development — meaning the same application code can point at whichever backend fits the deployment target without a rewrite. That abstraction is also the reason Semantic Kernel is the natural bridge into Azure, which I'll get to.

None of these frameworks ship a semantic cache out of the box, which is deliberate — caching is a cross-cutting, infrastructure-layer concern that doesn't belong bolted onto an agent framework's memory API. It belongs in front of the model call, which is usually a gateway, a dedicated cache client, or (on Azure) an API Management policy.

How do I add semantic caching to any of these frameworks?

The cleanest integration point is wherever the LLM client is instantiated, independent of which agent framework is calling it — which is exactly why LangChain exposes a global cache hook rather than a framework-specific one.

from langchain_redis import RedisSemanticCache
from langchain.globals import set_llm_cache
from langchain_openai import OpenAIEmbeddings

set_llm_cache(RedisSemanticCache(
    redis_url="redis://cache.internal:6379",
    embeddings=OpenAIEmbeddings(model="text-embedding-3-small"),
    distance_threshold=0.08,   # tune against a labeled eval set — never ship the default blind
))

GPTCache takes a similar wrapping approach around the raw OpenAI client if you're not on LangChain at all. Either way, the threshold is the whole ballgame — start conservative, and validate it the same way you'd validate a ranking model: a labeled set of query pairs that should hit and pairs that should deliberately miss, checked before every threshold change ships, not eyeballed once and left alone.

What does agent memory and caching look like on Azure specifically?

Azure's story is the most explicitly layered of the three clouds, which makes it easier to reason about once you see the pieces. Azure AI Foundry Agent Service manages conversation threads server-side — the short-term, working-memory layer — so the agent doesn't need to reimplement session state itself. Semantic Kernel's VectorStore abstraction plugs into Azure AI Search or Cosmos DB for NoSQL (which added native vector search) for the long-term, semantic/episodic memory layer — the facts and history that need to survive across threads. And Azure API Management sits in front of Azure OpenAI as a genuinely separate caching layer, using its azure-openai-semantic-cache-lookup and azure-openai-semantic-cache-store policies backed by Azure Managed Redis with the RediSearch module enabled.

<!-- inbound: check cache before the request reaches Azure OpenAI -->
<azure-openai-semantic-cache-lookup
    score-threshold="0.05"
    embeddings-backend-id="openai-embeddings"
    embeddings-backend-auth="system-assigned" />

<!-- outbound: store the response for future similarity matches -->
<azure-openai-semantic-cache-store duration="300" />

The three pieces are independently swappable, which is the point of building them as separate layers rather than one monolithic "AI memory" product: you can run the Foundry Agent Service without APIM in front of it, or add semantic caching to a completely different backend later, without touching the memory layer at all.

flowchart TB
    Client(["Client"]) --> APIM["Azure API Management
semantic-cache-lookup"]
    APIM -->|"cache miss"| Foundry["Azure AI Foundry
Agent Service — threads"]
    Foundry --> SK["Semantic Kernel
VectorStore abstraction"]
    SK --> Search["Azure AI Search or
Cosmos DB NoSQL vector"]
    Foundry --> AOAI["Azure OpenAI"]
    AOAI --> APIM2["Azure API Management
semantic-cache-store"]
    APIM2 --> Client
    Redis[("Azure Managed Redis
+ RediSearch")] -.backs.-> APIM
    Redis -.backs.-> APIM2
          

What about AWS and GCP — do they have a caching layer too?

My earlier piece covered Bedrock AgentCore Memory and Vertex AI Memory Bank in depth — those are the memory-layer services. What that article didn't cover, because it's a genuinely separate feature, is that both clouds also ship prefix caching, and it's easy to mistake it for the same thing. Amazon Bedrock prompt caching is GA on Claude 3.5 Haiku, Claude 3.7 Sonnet, and the Nova model family, cutting input-token cost by up to 90% and latency by up to 85% on the cached portion of a prompt; as of January 2026 it also supports an optional one-hour cache TTL (up from the five-minute default) for Claude Sonnet 4.5, Haiku 4.5, and Opus 4.5, which matters for agents that reuse a large tool-definition or system-prompt prefix across a long-running session. Vertex AI context caching works the same way — a cached prefix bills at roughly 10% of the standard input rate on a hit.

# Bedrock: mark a cache breakpoint after the (large, repeated) system prompt
messages = [
    {"role": "system", "content": [
        {"text": system_prompt},
        {"cachePoint": {"type": "default"}},
    ]},
    {"role": "user", "content": [{"text": user_query}]},
]

Both are prefix caching, not semantic caching — exact-match, deterministic, and safe by construction. If you want true similarity-based semantic caching on AWS or GCP, that's a bring-your-own layer (GPTCache or a self-managed Redis instance), the same as it is for LangGraph and CrewAI. Neither cloud bundles it the way Azure's APIM policies do, which is a genuine differentiator worth knowing before you assume feature parity across providers.

Best practices for wiring this into a real system

Compose the three layers, don't conflate them. A production agent needs a checkpointer for turn-taking, a store for cross-session facts, and — separately, optionally — a semantic cache in front of the model call. Treating any two of those as interchangeable is exactly how you end up with an agent that "has memory" but still calls the LLM on every repeat question, or a cache that "remembers the user" but actually doesn't.

Give cache and memory separate invalidation policies. A memory TTL is about fact staleness — how long is "user prefers dark mode" still true? A cache TTL is about answer freshness — how long is a cached response still the right thing to say? These decay at different rates for different reasons, and reusing one TTL scheme for both means one of them is wrong most of the time.

Scope the cache key the same way you'd scope memory. Cross-tenant memory leakage gets discussed constantly; cross-tenant cache leakage almost never does, and it's the same bug. If your semantic cache key doesn't include a tenant or user identifier where the answer is genuinely user-specific, a similarity hit can return one customer's account details to another customer who happened to phrase a question similarly. Scope by workspace or user from day one, exactly as you would for a memory store.

Test thresholds like a ranking problem, not a config value. Build a labeled set of query pairs — "should hit," "should miss," and the genuinely tricky near-misses in between — and run it against every threshold change before it ships. The evaluation discipline I wrote about for agents generally applies directly here: a threshold that "seems fine" in a demo is not evidence, and cache incoherence is exactly the kind of bug that doesn't show up until real, varied traffic hits it.

Use the abstraction layer your framework already gives you. LangGraph's Store and Semantic Kernel's VectorStore both exist specifically so a backend swap — moving from a self-hosted Postgres store to a managed Cosmos DB instance, say — is a configuration change, not a rewrite. If you're calling a vector database's SDK directly from application code instead of through the framework's abstraction, you've opted out of that flexibility for no real benefit.

If you're exposing memory or cache stores to an agent as tools rather than baking them into the framework, wire them through MCP. I covered the mechanics of that in Building MCP Servers — the same standardized tool-calling interface that works for any external system works cleanly for a memory or cache backend, and it keeps the integration swappable at the protocol level, not just the SDK level.

What to carry away

Cache and memory are not two names for the same idea — cache decides whether to call the model at all, memory decides what goes into the call once you've decided to make it. And within caching, prefix caching and semantic caching are not the same mechanism wearing different provider names — one is an exact, safe, deterministic optimization, and the other is a probabilistic similarity match that can be silently wrong if you don't tune and test the threshold the way you'd test any ranking system. Get both distinctions straight before you touch a framework's memory API or a cloud's caching policy, because every integration decision downstream — which LangGraph primitive to use, whether Semantic Kernel's VectorStore is enough on its own, whether Azure's APIM policy or a self-hosted Redis cache is the right call — is easy once the mental model is right and genuinely confusing when it isn't.