Fit a context window

Learn it

Words to know

context window

Everything a model can read and write in one request, measured in tokens. It has a fixed maximum size.

token

The unit a model reads and writes: a word, a piece of a word, or a punctuation mark. Limits and prices are counted in tokens.

token budget

The number of tokens you allow one part of a prompt to use.

chunk

A piece of a document, small enough to search and to fit in a prompt.

How it works

1. The budget for chunks is windowSize - reserved. If that is below 0, the budget is 0.
2. Go through the chunks in the order given. That order is their priority: the most useful comes first.
3. If a chunk's tokens fit in what remains, take it and subtract its tokens.
4. If it does not fit, skip it and keep going. A later, smaller chunk may still fit.

Worked example

Input
chunks = [{ id: "a", tokens: 8 }, { id: "b", tokens: 3 }, { id: "c", tokens: 2 }]
windowSize = 10
reserved = 5

Output
["b", "c"]

The budget is 10 - 5 = 5. Skipping a does not end the loop: the two smaller chunks after it fill the budget exactly.

Why it matters

More context is not better by default: text that does not help costs money and distracts the model. A budget forces a choice, and the order of the chunks decides who wins.

Your turn

Choose which chunks go into the context window, in priority order, without going over the token budget.

chunks

a list in priority order, each with an id and its size in tokens, a whole number of 0 or more

windowSize

the size of the context window, in tokens; a whole number of 0 or more

reserved

tokens set aside for the instructions and the answer; 0 or more, and it may be larger than windowSize

Returns

The ids of the chunks that were taken, in the order they were given.

Input
chunks = [{ id: "a", tokens: 8 }, { id: "b", tokens: 3 }, { id: "c", tokens: 2 }]
windowSize = 10
reserved = 5

Output
["b", "c"]

Input
chunks = [{ id: "a", tokens: 5 }, { id: "b", tokens: 1 }]
windowSize = 5
reserved = 0

Output
["a"]

Input
chunks = [{ id: "a", tokens: 1 }]
windowSize = 4
reserved = 5

Output
[]

The checks also try

• no chunks at all
• three chunks where only two fit
• a chunk of 0 tokens and a budget of 0
• more reserved than the window, and a chunk of 0 tokens
• chunks that are not in order of size
• a chunk of 0 tokens after the budget is used up

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Work out the budget once, before the loop. After that, one number changes as you go: how much of it remains.

2. Hint 2: The key step

   For each chunk, compare its tokens with what remains. Only when it fits do you add its id and reduce what remains.

3. Hint 3: The outline

   Set remaining to windowSize - reserved, or to 0 if that is negative. For each chunk in order: if its tokens are at most remaining, add its id to the result and subtract its tokens from remaining. Return the result.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. The budget for chunks, never below zero. Reserving first guarantees room for the answer. When more is reserved than the window holds, the budget is 0, not a negative number.
2. Take a chunk only if it fits in what remains. A chunk that is exactly as large as what remains still fits, so the comparison is at most, not less than.
3. What remains shrinks by the chunk that was taken. Without the subtraction every chunk would be compared with the full budget, and the total could go far over the window.

The same numbered steps are the comments in the solution.

function solve(chunks, windowSize, reserved) {
  // 1. The budget for chunks, never below zero
  let remaining = Math.max(0, windowSize - reserved);
  const taken = [];
  for (const chunk of chunks) {
    // 2. Take a chunk only if it fits in what remains
    if (chunk.tokens <= remaining) {
      taken.push(chunk.id);
      // 3. What remains shrinks by the chunk that was taken
      remaining -= chunk.tokens;
    }
  }
  return taken;
}

def solve(chunks, windowSize, reserved):
    # 1. The budget for chunks, never below zero
    remaining = max(0, windowSize - reserved)
    taken = []
    for chunk in chunks:
        # 2. Take a chunk only if it fits in what remains
        if chunk["tokens"] <= remaining:
            taken.append(chunk["id"])
            # 3. What remains shrinks by the chunk that was taken
            remaining -= chunk["tokens"]
    return taken

Watch out

• Do not stop at the first chunk that is too large. Skip it and go on: the task is to take every chunk that still fits, and smaller chunks may follow.
• Use at most, not less than. A chunk of exactly the remaining size fits.
• Token counts are given here. In a real system they come from the model's tokenizer; counting characters or words gives the wrong budget.

Check yourself