Large Language Models (LLMs) have transformed how we build software, but they come with inherent challenges: high inference costs, variable latency, and non-deterministic outputs. For production-grade applications, relying solely on real-time API calls is often unsustainable. This is where AI Caching becomes a critical component of your LLMOps stack. Unlike traditional caching, which deals with static key-value pairs, AI caching requires handling semantic similarity, context variations, and the unique properties of generative outputs.
In this post, we’ll explore the different types of AI caching, when to use them, and how to implement robust caching strategies that balance cost efficiency with output quality.
Why AI Caching is Different
Traditional web caching works on exact matches. If a user requests /api/products/123, you cache the JSON response. If they request /api/products/123?sort=date, it’s a different cache key. AI models, however, operate on natural language. A user asking "What is the capital of France?" and "Where is Paris located in terms of country?" are semantically identical, but syntactically different.
Therefore, effective AI caching must address two distinct problems:
- Prompt Caching: Avoiding redundant API calls for semantically similar prompts.
- Response Caching: Storing and retrieving generated outputs to reduce latency for repeated queries.
Types of AI Caching Strategies
1. Exact Match Caching
The simplest form of caching involves hashing the prompt string. If the exact same prompt is sent again, return the cached response. This is low-cost and easy to implement but has low hit rates in dynamic conversational contexts.
2. Semantic (Vector) Caching
This approach uses embeddings to find "near-matches" to previous queries. If a new prompt is semantically similar to a cached one (based on cosine similarity), you return the cached response. This significantly increases hit rates but introduces the risk of serving slightly outdated or contextually incorrect answers.
3. Hybrid Caching
Most production systems use a hybrid approach: check for an exact match first (high confidence), then fall back to semantic matching (higher cost, lower confidence).
Implementation Example: Semantic Caching with Python
Below is a simplified example of how you might implement a semantic cache using numpy for cosine similarity and a vector store like ChromaDB or Pinecone. For brevity, we’ll use a local in-memory approach with embeddings generated by an OpenAI-compatible API.
import hashlib
import numpy as np
from typing import Dict, Tuple
import time
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.95):
self.cache: Dict[str, Dict] = {} # hash: {prompt, response, embedding, timestamp}
self.threshold = similarity_threshold
def _get_embedding(self, text: str) -> np.ndarray:
# Placeholder: In production, use your LLM's embedding API
# e.g., openai.embedding.create(input=text)
# Here we simulate a deterministic embedding for demonstration
# In real code, this calls an embedding service
dummy_hash = hashlib.md5(text.encode()).hexdigest()
# Simulate a 1024-dimensional vector
np.random.seed(int(dummy_hash, 16))
return np.random.rand(1024)
def _cosine_similarity(self, vec1: np.ndarray, vec2: np.ndarray) -> float:
dot_product = np.dot(vec1, vec2)
norms = np.linalg.norm(vec1) * np.linalg.norm(vec2)
if norms == 0:
return 0
return float(dot_product / norms)
def get(self, prompt: str) -> Tuple[bool, str]:
prompt_hash = hashlib.md5(prompt.encode()).hexdigest()
# 1. Check Exact Match
if prompt_hash in self.cache:
return True, self.cache[prompt_hash]['response']
# 2. Check Semantic Match
current_embedding = self._get_embedding(prompt)
best_match_score = 0
best_match_response = None
for hash_key, data in self.cache.items():
cached_embedding = data['embedding']
similarity = self._cosine_similarity(current_embedding, cached_embedding)
if similarity > best_match_score:
best_match_score = similarity
best_match_response = data['response']
if best_match_score >= self.threshold:
return True, best_match_response
return False, ""
def put(self, prompt: str, response: str):
prompt_hash = hashlib.md5(prompt.encode()).hexdigest()
embedding = self._get_embedding(prompt)
self.cache[prompt_hash] = {
'prompt': prompt,
'response': response,
'embedding': embedding,
'timestamp': time.time()
}
# Usage Example
if __name__ == "__main__":
cache = SemanticCache(similarity_threshold=0.90)
# First call: Cache miss
hit, resp = cache.get("What is the capital of France?")
print(f"Call 1: Hit={hit}") # Expected: False
# Simulate LLM response
llm_response = "The capital of France is Paris."
cache.put("What is the capital of France?", llm_response)
# Second call: Exact match
hit, resp = cache.get("What is the capital of France?")
print(f"Call 2: Hit={hit}, Response={resp}") # Expected: True, Paris
# Third call: Semantic match
hit, resp = cache.get("Where is Paris located in terms of country?")
# Note: With real embeddings, this would likely be a hit if similarity > 0.90
print(f"Call 3: Hit={hit}, Response={resp}")
Best Practices and Pitfalls
- Cache Invalidation: LLM knowledge can become outdated. Implement TTL (Time-To-Live) policies or manual invalidation triggers when your knowledge base updates.
- Context Sensitivity: Be cautious with semantic caching in multi-turn conversations. A similar question might have a different answer depending on previous context. Consider including conversation history in the cache key or disabling semantic caching for stateful chats.
- Threshold Tuning: The similarity threshold is a critical hyperparameter. Too low, and you risk serving incorrect or biased responses. Too high, and your cache hit rate drops. Start with 0.95+ and monitor user feedback.
- Logging and Monitoring: Track cache hits/misses, similarity scores, and user-reported errors on cached responses. This data is essential for tuning your strategy.
Conclusion
AI Caching is not just a performance optimization; it’s a strategic lever for controlling costs and improving user experience in LLM applications. By combining exact match and semantic caching with robust monitoring, you can significantly reduce latency and API spend without sacrificing the quality of your AI outputs. Start small with exact matching, then introduce semantic caching with careful threshold tuning. As your LLMOps maturity grows, these techniques will become foundational to building scalable, cost-effective AI systems.