Retrieval-Augmented Generation (RAG) has become the standard architecture for building production-ready Large Language Model (LLM) applications. By grounding generative models in external knowledge bases, organizations can mitigate hallucinations and provide citations. However, a critical bottleneck remains: the quality of the retrieval step. If the initial query fails to match the semantic meaning of the stored documents, even the most powerful LLM will produce poor results.
Enter Query Expansion. This technique involves transforming a user's sparse, often ambiguous input into a richer, more comprehensive representation before sending it to the vector database. By expanding the query, we increase the probability of finding relevant context, thereby improving the overall quality of the generation phase.
Why Simple Vector Search Fails
Consider a user asking: "My laptop won't turn on." A naive semantic search might look for documents containing those exact words or close vector neighbors. However, the relevant troubleshooting guide in your knowledge base might be titled: "Troubleshooting Power Issues on Device Boot Failure".
In this scenario, the lexical and semantic gap between the user's informal query and the formal document title causes retrieval failure. Query expansion bridges this gap by generating multiple versions of the query, each focusing on different aspects or synonyms.
Common Query Expansion Strategies
There are three primary methods for implementing query expansion in a RAG pipeline:
1. Multi-Vector Expansion (RQ-RAG)
This approach uses a dedicated model to decompose a single query into multiple semantic vectors. Each vector captures a different perspective of the user's intent. When searching, the system performs a similarity search for all expanded vectors and merges the results.
2. LLM-Based Synthetic Expansion
Here, an LLM is prompted to rewrite the original query. It might generate synonyms, related questions, or a broader statement. This is highly flexible but introduces latency and cost.
3. Keyword & Hyphenated Expansion
Combining dense vector search with sparse lexical search (like BM25). By extracting keywords from the query and searching for them alongside the vector representation, you capture exact matches that semantic search might miss.
Implementing LLM-Based Query Expansion
Below is a practical Python example using LangChain and an LLM to generate multiple query variations. This technique is often referred to as "HyDE" (Hypothetical Document Embeddings) or simply multi-query retrieval.
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
# Define the prompt for generating related queries
prompt_template = """You are an AI assistant tasked with improving search results.
Given the following user query, generate 3 distinct variations that might retrieve more relevant documents.
Focus on synonyms, related technical terms, and different phrasing.
User Query: "{query}"
Generated Variations:
1.
2.
3.
"""
llm_chain = LLMChain(
llm=OpenAI(temperature=0),
prompt=PromptTemplate(
input_variables=["query"],
template=prompt_template
)
)
# Example usage
original_query = "How to fix database connection timeout in Python?"
response = llm_chain.run(original_query)
# In a real pipeline, you would parse the response into a list of strings
# and pass those strings to your vector store retriever.
print(response)Best Practices and Trade-offs
While query expansion significantly improves recall, it comes with trade-offs. Latency increases because you are making additional API calls to the LLM and performing multiple database lookups. Cost also rises due to higher token consumption.
To mitigate these issues, consider the following best practices:
- Caching: Cache expanded queries to avoid redundant computation for common search terms.
- Result Reranking: Use a cross-encoder reranker on the top-K results from all expanded queries to ensure the final context is highly relevant.
- Hybrid Search: Combine query expansion with BM25 keyword matching to capture both semantic intent and exact terminology.
Conclusion
Query expansion is not just a nice-to-have feature; it is a fundamental component of robust RAG systems. By acknowledging that user queries are often sparse and ambiguous, developers can implement expansion strategies that bridge the gap between human intent and machine retrieval. Whether you choose multi-vector decomposition or LLM-driven synthetic queries, the goal remains the same: to provide the LLM with the precise context it needs to generate accurate, helpful, and grounded responses.
As RAG architectures evolve, integrating dynamic query expansion will separate production-grade applications from experimental prototypes. Start simple, measure your retrieval metrics, and iterate.