Gains:
- Numerically evaluate chunk size, overlap and semantic chunking tradeoffs
- Choosing appropriate chunking strategy for different document types (PDF, table, code, chat log)
- Strengthen retrieval quality and filtering by adding metadata to each chunk
This is the most overlooked but most decisive step in RAG: how you break down the document. This is called chunking. Even if you give the same document to the same model, due to bad chunking the retrieval returns the wrong piece and the model will never produce a great answer. In this unit, we cover fragmentation strategies, how to adapt them according to document type, and how to add meaningful metadata to each fragment.
Why Do We Shred?
There are three reasons. First, embedding models convert text up to a certain length into a meaningful vector; If an entire 40-page chapter is crammed into a single vector, the meaning becomes "blurred." Second, we want to give the model only the part that is needed as context; handing over the entire document is expensive and distracting. Third, for retrieval to be precise, the search unit must be small and focused.
So chunk is the smallest unit of retrieval. It should not be too big or too small – just right.
Chunk Size and Overlap Balance
There are two main settings: chunk size (how many tokens/words will be in a chunk) and overlap (the portion shared by neighboring chunks).
Very small chunks (e.g. 100 tokens): focused but disconnected from context. He says "for 14 days", but what 14 days is is left in the previous sentence. Very large chunks (e.g. 2000 tokens): preserves context but many threads are mixed in; embedding gets muddled and irrelevant topics come together.
Overlap solves the boundary problem. If a sentence falls exactly on the border of two parts, it is divided into two without overlapping and its meaning is lost. Overlap of 50-100 tokens ensures that the information falling within the limit remains intact in at least one part.
Chunk size
Advantage
Disadvantage
appropriate content
Small (100-250 tokens)
High sensitivity, focused
Context may break
FAQ, short articles, definitions
Medium (300-600 tokens)
Balance; most scenarios
—
Procedures, policy texts
Large (800-1500 tokens)
Context integrity
blurry embedding
Narrative, long explanations
Tip: If you don't know where to start, start with 400-500 token chunk and 50-80 token overlap; then measure and adjust with your own data. The "right" size is not universal, it depends on the context.
Chunking Strategies
Fixed-size: Trims the text every N tokens. It's simple and fast, but can interrupt mid-sentence.
Separator-based (recursive/separator): Divides according to paragraph and then sentence boundaries; It better preserves the integrity of meaning. Most production systems start with this.
Semantic chunking: It looks at the embeddings of the sentences and divides them where the subject change occurs. It is the highest quality but most expensive method; With large volumes, transaction costs increase.
Structure-aware: Uses document structure such as headings, sections, tables. For example, splitting a Markdown document by headings ensures that each part carries its own heading.
Adaptation by Document Type
Not every document is the same. Strategy varies by type:
- PDF/policy text: Bookmark-based, medium size. Clear page top/bottom repeats (header/footer).
- Tables: Do not take the line out of context; keep each row with header information ("Item: X, Price: Y, Stock: Z"). Converting the raw table to plain text is often essential.
- Code: Split by function/class boundaries; Don't cut a function out of the way.
- Chat/ticket recording: Split by message or conversation round; Maintain knowledge of who said what.
# bracket-based chunking (conceptual) chunks = bol( text, target_size=450, # token overlap=70, # token brackets=["\n\n", "\n", ". ", " "] # paragraph first, word last)
Add Metadata to Each Track
Chunking is not just "divide"; is to enrich each piece. Every tag you attach to the track is worth its weight in gold for future filtering and source citing.
# Enriched chunk (conceptual){ "text": "Annual paid leave is 14 days with 1-5 years of service...", "metadata": { "source": "ik_el_kitabi_v7.pdf", "section": "5.2 Annual Leave", "page": 23, "date": "2025-06", "department": "IK", "privacy": "internal" }}
Another powerful technique is adding a contextual header: writing the title of the chapter it belongs to at the beginning of each piece. Thus, even a disjointed piece like "For 14 days" is both better embedded and more meaningful as "Annual Leave - 14 days."
Weak Chunking / Strong Chunking
Weak (blind hardcut, no metadata):
Truncate text every 1000 characters. Keep only the text.# Result: tables are split in the middle, "14 days" remains without context,# it is not known which document it came from, no filter can be made.
Powerful (structure-aware + header + metadata):
Divide the document by headings; add section title to each part;attach source, page, date and privacy metadata; convert tablerows to plain text with their headers.# Result: focused, contextual, filterable, sourceable.
Three Mini Cases
Case 1 — Painting disaster. A finance team divided the 200-page price list with blind hard cutting; table rows were randomly split. "What is the price of product X?" The model read the wrong line and gave the wrong price (9 out of 12 cases are wrong). When I converted the table rows to plain text in the format "Product: … | Price: … | Unit: …" the error decreased to 0 out of 12.
Case 2 — Extremely large chunk. In a wiki, each page is made of a single chunk (some say 3,000 tokens). The embedding is blurred because there is "leave", "overtime" and "payroll" on one page; The working hours section also came into play regarding the leave question. When the pages were divided into medium size by title, recall@5 increased from 64% to 91%.
Case 3 — Truncated sentence without overlap. 250 token fixed cut for a legal team, no overlap. A critical definition fell right on the border of two parts and split in two; Neither one nor the other contains the complete answer. When 60 token overlap was added, the same definition remained intact in one piece and the correct answer was returned.
Common mistakes
- Blind fixed cut: Splits sentences and tables in the middle; meaning is lost.
- Leaving the overlap at zero: Information that falls on the boundary is divided and lost.
- Not adding metadata: Filtering and source display become impossible.
- Leaving tables raw: The model cannot resolve the table structure; Convert lines to plain text.
- Imposing one strategy: PDF, code and table are not split by the same method; Adapt to genre.
Caution: Don't set Chunking once and forget it. Re-measure retrieval quality as new document types arrive (tickets from a new system, scanned PDFs). Bad input data means bad response ("garbage in, garbage out").
In summary
- Chunk is the smallest unit of retrieval; Neither too big nor too small – it should be balanced according to the content.
- Chunk size indicates focus-context balance; Overlap manages boundary loss.
- Bracket-based and structure-aware chunking is the starting point of most generation systems; semantic chunking is good quality but expensive.
- Types like table, script, and chat require their own strategies; Convert tables to plain text.
- Add source/date/chapter/privacy metadata and section title to each track; This is the basis of filtering and citation.
Application task
Break a section of the document you choose into three different ways: (1) small pieces of 200 tokens, (2) medium pieces of 500 tokens (70 token overlap), (3) single large pieces. Ask the same 3 questions for each strategy, manually mark which piece to bring in, and write down the reasoning for which strategy works best for that document. Then add at least four metadata fields and a “chapter title” to each track. If the document contains a table, convert a table row to plain text in the "field:value" format.
checklist
- [ ] I can tell that chunk is the smallest unit of retrieval and size is the focus-context balance.
- [ ] I know why Overlap prevents boundary loss.
- [ ] I can distinguish between bracket-based, semantic and structure-aware chunking.
- [ ] I can adapt the strategy for table, code and chat.
- [ ] I reinforce retrieval by adding metadata and chapter title to each track.