Split before making windows

Learn it

Words to know

validation set

Examples kept out of training and used only to check how the model does on data it has not seen.

data leakage

When information from the validation data gets into training, so the model looks better than it is.

context window

The number of tokens the model reads at one time.

How it works

1. Split the token list at cut: positions before cut are training, the rest are validation.
2. Inside each part, take every stretch of context + 1 tokens, moving the start one position at a time. The extra token is the last target.
3. No window may cross the cut. A part with N tokens gives N - context windows, or none if that is 0 or less.

Worked example

Input
tokens = [0, 1, 2, 3, 4, 5, 6]
cut = 4
context = 2

Output
[[[0, 1, 2], [1, 2, 3]], [[4, 5, 6]]]

The stretches [2, 3, 4] and [3, 4, 5] are left out on purpose. They would cross the cut and put validation tokens into training.

Why it matters

The validation loss from chapter 4 only means something if the split is clean. The workshop's GPT splits its text this way, so its validation loss is measured on sentences it has not trained on.

Your turn

Split a token sequence into training and validation parts, then list every full window inside each part.

tokens

the token IDs of the whole text

cut

the position where validation begins; between 0 and the length of tokens

context

the model's window length; positive

Returns

The list [train_windows, validation_windows]. Each window has context + 1 tokens, windows start one position apart, and none crosses cut.

Input
tokens = [0, 1, 2, 3, 4, 5, 6]
cut = 4
context = 2

Output
[[[0, 1, 2], [1, 2, 3]], [[4, 5, 6]]]

Input
tokens = [0, 1, 2, 3, 4, 5]
cut = 3
context = 2

Output
[[[0, 1, 2]], [[3, 4, 5]]]

Input
tokens = [0, 1]
cut = 1
context = 2

Output
[[], []]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Split first: tokens[:cut] and tokens[cut:]. Then do the same job on each part, so write that job as a small helper function.

2. Hint 2: The key step

   In a part of length N, a window of context + 1 tokens can start at positions 0 to N - context - 1. That is range(N - context), which is empty when the part is too short.

3. Hint 3: The outline

   Define windows(part) returning part[i : i + context + 1] for each i in range(len(part) - context). Return [windows(tokens[:cut]), windows(tokens[cut:])].

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Split the text before making any window. After the split, the two parts share no token. Whatever is cut from one cannot overlap the other.
2. Every stretch of context + 1 tokens inside one part. The extra token is the target for the last position. range of 0 or a negative number is empty, so a short part gives no windows and no error.

The same numbered steps are the comments in the solution.

def solve(tokens, cut, context):
    # 1. Split the text before making any window
    train = tokens[:cut]
    validation = tokens[cut:]

    # 2. Every stretch of context + 1 tokens inside one part
    def windows(part):
        return [part[i : i + context + 1] for i in range(len(part) - context)]

    return [windows(train), windows(validation)]

Watch out

• Do not make the windows first and split the list of windows. Neighbouring windows share all but one token, so the two sides would overlap.
• A window has context + 1 tokens, not context: the input and one more token for the final target.

Check yourself