chunking

In depth

The trade-off

The trade-off is easy to state and hard to settle. Short fragments give a search that is precise but amputated: the clause comes back without the article that conditions it, the table without its caption, the answer without the question. Long fragments preserve the context but dilute the meaning: their representation becomes a blurred average that no longer resembles anything, and the search misses them. No universal size exists, because the right unit depends on the structure of your documents.

Cutting by structure

The robust solution consists in cutting according to structure rather than by the metre. A contract is cut by article, a manual by section, minutes by agenda item, a web page by heading level. Two additions do the rest: an overlap between neighbouring fragments, so that a sentence straddling both is lost by neither, and a reminder header on each fragment, carrying the title of the document and the path of the section, so that an isolated fragment stays readable.

The case of the table

One frequent trap deserves to be named: the table. Cut down the middle, it loses its headers and becomes a series of numbers with no meaning, which the model will interpret confidently. Composite documents, alternating text, tables and images, therefore call for preliminary processing that recognises these blocks and keeps them whole. It is a job of document preparation, often the longest in an internal search project, and it cannot be delegated to a generic library.