Glossary
What is Chunking strategy?
Last updated: 2026-05-26
Definition
A chunking strategy is the algorithm a RAG pipeline uses to split source documents into the pieces ("chunks") it will later embed, store, and retrieve. Chunking is the single biggest determinant of RAG quality: poor chunking puts the right answer in the wrong slice and no embedding model can recover it. There is no universal optimum — strategy must match document type, query pattern, and embedding model context window.
Why Chunking strategy matters
Practitioners spend more time tuning chunking than any other RAG knob. Recent benchmarks show 20-40% retrieval accuracy swing between naive fixed-size chunking and structure-aware chunking on the same corpus. For regulated content (contracts, policies, medical records) chunking errors can leak personal data across context boundaries or split a clause from its definition — both are compliance issues.
How Chunking strategy works
- 1Fixed-size chunking: split every N tokens (typical 256-1024) with M-token overlap (typical 10-20%). Fast, dumb, often good enough.
- 2Sentence- or paragraph-aware: split at natural prose boundaries; better for narrative text but breaks on tables, code, lists.
- 3Structure-aware: parse the document (markdown headings, HTML sections, PDF outline) and chunk per logical unit. Best for technical docs and contracts.
- 4Semantic chunking: use embedding similarity to detect topic shifts and split at them. Slow to build, strong for long-form essays.
- 5Hybrid retrieval: store the same content at multiple chunk sizes ("parent-child") and retrieve the small chunk first, then expand to its parent for context.
Examples
- A policy library uses heading-aware chunking: every H2 becomes a chunk, and the H1 + breadcrumb is prepended so each chunk knows which policy it belongs to.
- A codebase RAG splits per function (tree-sitter parse), not per line — so when a developer asks "how does login() work?" the retrieval returns the full function.
- A customer-support archive uses semantic chunking on conversation transcripts to keep each "topic shift" as its own chunk for higher-precision retrieval.
References
Related concepts
Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is a technique that grounds a large language model in a specific corpus of documents at query time. Instead of relying only on what the model learned during training, RAG retrieves relevant passages from your data and adds them to the prompt — letting the model answer using your knowledge, current and proprietary.
AI agent
An AI agent is a software program that uses a large language model (LLM) to autonomously plan and complete a task, combining reasoning, tool use, and memory. Unlike a one-shot prompt, an agent can break a goal into steps, call external tools or APIs, and decide what to do next based on intermediate results.
Model Context Protocol (MCP)
Model Context Protocol (MCP) is an open standard introduced by Anthropic in 2024 that defines how AI agents connect to external data sources and tools. MCP servers expose data and capabilities; MCP clients (LLMs and agent platforms) discover and call them through a uniform interface — eliminating per-tool custom integration code.
Multi-LLM chat
Multi-LLM chat is a chat interface that lets you switch between multiple large language model vendors — OpenAI (GPT), Anthropic (Claude), Google (Gemini), Mistral, and others — inside a single conversation thread. You pick the model best suited to the next turn instead of being locked into one vendor for the whole task.
FAQ
Chunking strategy — common questions
- What is the optimal chunk size?
- There is no universal optimum. Starting point: 512 tokens with 10% overlap for prose, structure-aware for technical docs, function-level for code. Iterate based on actual retrieval evals against your query set.
- Should chunks contain metadata?
- Yes — at minimum: source URL, section path, last_modified, access level. This lets retrieval filter ("only this customer's docs"), let answers cite sources, and lets compliance audit which document version was used.
- Do I need to re-chunk when I switch embedding models?
- Usually yes. Different embedding models have different optimal chunk sizes (driven by context length and training distribution). Always re-evaluate retrieval quality after an embedding-model swap.