Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding Large Language Models in proprietary data. However, the quality of the generation is inextricably linked to the quality of the retrieval. If the system retrieves irrelevant, fragmented, or incomplete context, the model will hallucinate or produce poor answers. The most critical, yet often overlooked, variable in this pipeline is chunking.
Chunking is the process of splitting large documents into smaller, manageable pieces that can be embedded into a vector space. While splitting text might seem like a trivial preprocessing step, it is actually a complex balancing act between information density, semantic coherence, and computational constraints. In this post, we explore advanced chunking strategies that move beyond naive character limits to create contextually rich fragments.
The Problem with Naive Fixed-Size Chunking
The simplest approach to chunking is fixed-size splitting. This involves slicing text into segments of $N$ characters (or tokens) with an optional overlap. While easy to implement, this method suffers from severe semantic fragmentation.
Consider a document explaining a complex legal clause. A fixed-size chunk might start in the middle of a sentence, end before the conclusion of a thought, or split a table across two chunks. When these fragments are embedded, the vector representation captures only partial meaning. During retrieval, if the user’s query relates to the concept spanning the split boundary, the retriever may fail to rank the relevant chunks highly because no single chunk contains the complete logical unit.
Recursive Character Splitting
A more robust baseline is Recursive Character Splitting. Instead of blindly cutting by characters, this method attempts to split text using natural delimiters in a hierarchy. The goal is to keep sentences, paragraphs, and sections intact wherever possible.
The algorithm typically operates by defining a list of separators (e.g., `"\n\n"`, `"\n"`, `" "`) and recursively splitting the text by the first separator. If the resulting chunks are still larger than the desired maximum, it moves to the next separator. This ensures that chunks align with natural linguistic boundaries.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Define the splitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
length_function=len,
is_separator_regex=False,
separators=["\n\n", "\n", " ", ""]
)
text = """
Section 1: Introduction
The history of AI is long.
Section 2: Current State
Modern LLMs rely on transformers."""
chunks = text_splitter.split_text(text)
# Resulting chunks respect paragraph breaks before splitting further
Semantic and Sentence-Window Chunking
For higher-end applications, Semantic Chunking leverages embeddings to determine where to split text. By calculating the cosine similarity between consecutive sentence embeddings, the algorithm identifies points where the topic changes significantly. A drop in similarity indicates a boundary between distinct concepts, allowing chunks to be formed around coherent themes.
Another powerful hybrid approach is Sentence-Window Chunking. In this strategy, the text is first split into sentences. However, when a sentence is selected as a "primary" chunk for embedding, it is accompanied by a context window of $K$ previous and $K$ next sentences. This provides the vector database with the local context needed to understand the primary sentence, while the LLM can use this window to answer questions that require slight context shifts.
Tuning Overlap and Granularity
Regardless of the strategy, overlap is a critical parameter. Overlap ensures that information at the boundaries of chunks is not lost. A typical overlap is 10-20% of the chunk size. Too little overlap risks missing bridging information; too much overlap increases storage costs and retrieval latency without significant gains in accuracy.
Additionally, granularity matters. Smaller chunks (e.g., 128-256 tokens) generally yield more precise retrieval because the embedding is focused on a narrower topic. Larger chunks (e.g., 512-1024 tokens) provide more context but may dilute the vector representation. It is best practice to experiment with both dimensions using your specific dataset and query set.
Conclusion
Effective chunking is not a one-size-fits-all solution. It requires understanding the structure of your source data and the nature of your user queries. Start with recursive character splitting for general purpose documents, but move to semantic or window-based strategies for technical or legal text where context continuity is paramount. By treating chunking as a core component of your RAG pipeline rather than a preprocessing afterthought, you significantly enhance the intelligence and reliability of your generated responses.