Traditional keyword-based search engines fail to understand context. If a user searches for "how to fix a broken pipe," a keyword search might miss documents containing "plumbing repair" or "leak resolution." This is where Semantic Search shines. By converting text into high-dimensional vector representations (embeddings), we can capture the underlying meaning of content rather than just matching exact strings. For developers building Retrieval-Augmented Generation (RAG) systems, semantic search is not just an optimization—it is the critical first step in retrieving relevant context for Large Language Models (LLMs).
How Semantic Search Works
Semantic search operates on the principle of cosine similarity or dot product distance between vectors. The process involves two main stages:
- Embedding Generation: Using a pre-trained transformer model (like Sentence-BERT or OpenAI's Embedding API), text chunks are converted into numerical vectors. Similar meanings result in vectors that are close together in the multi-dimensional space.
- Vector Retrieval: When a query is made, it is also embedded. The system then searches a vector index to find the $K$ nearest neighbors (KNN) to the query vector.
Implementing Semantic Search with FAISS
Facebook AI Similarity Search (FAISS) is a popular library for efficient similarity search on dense vectors. Below is a practical example of setting up a semantic search pipeline using Python, FAISS, and Hugging Face's transformers.
import faiss
import numpy as np
from transformers import AutoTokenizer, AutoModel
def get_sentence_embeddings(sentences, model_name='sentence-transformers/all-MiniLM-L6-v2'):
"""Generate normalized embeddings for a list of sentences."""
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
output = model(**inputs)
embeddings = output.last_hidden_state[:, 0, :] # [CLS] token
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
return embeddings.cpu().numpy()
# 1. Prepare Data
documents = [
"The sky is blue during the day.",
"Python is a popular programming language.",
"Machine learning models require large datasets.",
"The ocean is vast and deep."
]
# 2. Generate Document Embeddings
doc_embeddings = get_sentence_embeddings(documents)
doc_embeddings = np.float32(doc_embeddings)
# 3. Build FAISS Index (Inner Product)
dimension = doc_embeddings.shape[1]
index = faiss.IndexFlatIP(dimension)
index.add(doc_embeddings)
# 4. Perform Search
query = "What is the weather like outside?"
query_embedding = get_sentence_embeddings([query])
query_embedding = np.float32(query_embedding)
# Search for top 2 relevant documents
k = 2
distances, indices = index.search(query_embedding, k)
for i, doc_idx in enumerate(indices[0]):
print(f"Score: {distances[0][i]:.4f} | Doc: {documents[doc_idx]}")
Best Practices for Production Systems
- Chunking Strategy: Break large documents into smaller, contextually complete chunks (e.g., 100-500 tokens) to improve embedding precision.
- Hybrid Search: Combine semantic search with BM25 (keyword search) for best-of-both-worlds results. This ensures that rare proper nouns or specific codes are still retrievable.
- Re-ranking: Use a cross-encoder model to re-rank the top $N$ candidates returned by the vector search to improve final accuracy.
Conclusion
Semantic search is the bridge between unstructured data and intelligent AI applications. By leveraging vector databases and robust embedding models, you can significantly enhance the performance of RAG pipelines, ensuring that your LLMs have access to the most relevant context possible. Start small with FAISS, but consider scaling to dedicated vector databases like Pinecone, Weaviate, or Milvus as your data volume grows.