Overlapping document chunks

Learn it

Words to know

chunk

A piece of a document, small enough to search and to fit in a prompt.

overlap

The words at the end of one chunk that are repeated at the start of the next.

RAG

Retrieval-augmented generation: find the documents that matter for a question, and give them to the model together with the question.

How it works

1. Split the text into words at whitespace.
2. The first chunk is the first size words: it starts at position 0.
3. Move the start forward by size - overlap words, so the next chunk begins with the last overlap words of the previous one.
4. Stop as soon as a chunk reaches the last word. Going on would only make chunks that repeat what is already covered.

Worked example

Input
text = "the cat sat on the red mat"
size = 4
overlap = 2

Output
["the cat sat on", "sat on the red", "the red mat"]

Each chunk starts 2 words after the previous one and shares 2 words with it. The last chunk is shorter because the text ran out. Positions count from 0 and include both ends.

Why it matters

Chunks are what a retrieval system stores, ranks and hands to the model. Chunk size and overlap change which facts can be found, so the roadmap asks you to measure them, not guess.

Your turn

Split a text into chunks of size words, where each chunk starts with the last overlap words of the one before it.

text

the document, as one string; words are separated by spaces, tabs or line breaks, and it may be empty or only spaces

size

how many words a full chunk holds; 1 or more

overlap

how many words a chunk shares with the one before it; from 0 to size - 1

Returns

A list of chunks in order. Each chunk is a string: its words joined by single spaces. The last chunk may be shorter, and no chunk follows the one that holds the last word. For a text with no words, an empty list.

Input
text = "the cat sat on the red mat"
size = 4
overlap = 2

Output
["the cat sat on", "sat on the red", "the red mat"]

Input
text = "a b c d e"
size = 3
overlap = 1

Output
["a b c", "c d e"]

Input
text = "a b c d e"
size = 2
overlap = 0

Output
["a b", "c d", "e"]

Input
text = "  "
size = 2
overlap = 0

Output
[]

The checks also try

• extra spaces and a line break
• one chunk holds everything
• fewer words than the size
• spaces at both ends, size 1
• punctuation and capital letters stay
• the same words again and again
• a tab between words

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   First turn the text into a list of words. Then every chunk is a slice of that list, joined back together with spaces.

2. Hint 2: The key step

   Keep a start position. A chunk is the words from start up to start + size. After each chunk, move start forward by size - overlap.

3. Hint 3: The outline

   Put every run of non-space characters in words. Set start to 0. While start is less than the number of words: take the words from start up to start + size, join them with a space, and add that chunk. If start + size is at least the number of words, stop. Otherwise add size - overlap to start.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Split the text into words. Splitting at any run of whitespace handles double spaces and line breaks. A text with no words gives an empty list, and the loop below never runs.
2. A chunk is the next size words, joined by spaces. Slicing past the end of a list is safe in both languages: it returns the words that are there, which makes the last chunk shorter.
3. Stop once a chunk has reached the last word. Without this the loop would go on and add chunks made only of words that the previous chunk already holds.
4. The next chunk starts size minus overlap words later. Moving by less than a full chunk is what makes two neighbours share their last and first words.

The same numbered steps are the comments in the solution.

function solve(text, size, overlap) {
  // 1. Split the text into words
  const words = text.match(/\S+/g) ?? [];
  const chunks = [];
  let start = 0;
  while (start < words.length) {
    // 2. A chunk is the next size words, joined by spaces
    chunks.push(words.slice(start, start + size).join(" "));
    // 3. Stop once a chunk has reached the last word
    if (start + size >= words.length) break;
    // 4. The next chunk starts size minus overlap words later
    start += size - overlap;
  }
  return chunks;
}

def solve(text, size, overlap):
    # 1. Split the text into words
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        # 2. A chunk is the next size words, joined by spaces
        chunks.append(" ".join(words[start : start + size]))
        # 3. Stop once a chunk has reached the last word
        if start + size >= len(words):
            break
        # 4. The next chunk starts size minus overlap words later
        start += size - overlap
    return chunks

Watch out

• Do not move the start by size. That makes chunks with no overlap at all. The step is size - overlap.
• Leaving out the early stop adds extra chunks at the end that hold only repeated words, such as a lone e.
• This lesson counts words. A real system counts tokens with the model's own tokenizer, and one word is often more than one token.
• In JavaScript: In JavaScript, " a b ".split(/\s+/) leaves an empty string at each end, and trimming first still gives [""] for a text of only spaces. Match the words instead: text.match(/\S+/g) ?? [].
• In Python: In Python, text.split() with no argument drops empty pieces. text.split(" ") does not: it leaves an empty string for every extra space.

Check yourself