Build GPT · 2 of 10 · Python

Create a reproducible character vocabulary

Before a model can read text, you must decide which tokens exist and give each a number. That list is the vocabulary. The simplest one treats every different character as a token. What matters is that the numbering is the same every time you build it, because a trained model has learned what each number means. Sorting the characters gives that: the same text always produces the same list in the same order.

Do the exercise in the app

Learn it

Words to know

vocabulary: The full list of tokens a model knows. A token's place in the list gives its ID.
token ID: The whole number that stands for a token.
token: One unit of text as the model sees it: a character, a piece of a word or a whole word.

How it works

Collect every different character in the text. A space is a character too.
Sort them. Python sorts characters by their code number, so a space comes before letters, and capitals come before lowercase.
The position in the sorted list decides the ID. The next lesson keeps ID 0 for unknown characters, so there the first listed character gets ID 1. The workshop's GPT has no unknown ID and counts from 0.

Worked example

Input
text = "hello"

Output
["e", "h", "l", "o"]

Step	Result
Characters in the text	h, e, l, l, o
Different characters	h, e, l, o
Sorted	e, h, l, o
IDs they will get	e=1, h=2, l=3, o=4

The letter l appears twice in the text but once in the vocabulary. The model will have 5 possible IDs: 4 characters and the unknown ID 0.

Why it matters

The workshop's GPT builds its vocabulary with this same line and saves it inside the checkpoint. The size of the vocabulary sets the size of the model's first and last layers.

Your turn

Build the character vocabulary of a text.

text: any string; may be empty
Returns: A sorted list of the different characters in text, each once. Whitespace characters are included.

Example 1

Input
text = "hello"

Output
["e", "h", "l", "o"]

Four different characters, sorted.

Example 2

Input
text = "baBa"

Output
["B", "a", "b"]

Capital B is a different character from b, and it sorts before the lowercase letters.

Example 3

Input
text = " a\n"

Output
["\n", " ", "a"]

The space and the line break are characters too, at the start or end as much as in the middle. Both sort before the letter.

Example 4

Input
text = ""

Output
[]

An empty text has no characters.

Hints

Each hint gives away a little more. Open one only when you need it.

Hint 1: Where to start
You need each character once, however many times it appears. Python has a data type that keeps one of each.
Hint 2: The key step
set(text) holds the different characters, in no particular order. Sorting puts them in a fixed order.
Hint 3: The outline
Return sorted(set(text)): the set drops the repeats, and sorted returns a list in a fixed order.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

A set keeps one of each character. Iterating over a string gives its characters one by one, and a set drops the repeats.
Sorting gives the same order every time. A set has no fixed order. Without sorting, the IDs could differ from one run to the next and a saved model would read nonsense.

The same numbered steps are the comments in the solution.

Python

def solve(text):
    # 1. A set keeps one of each character
    characters = set(text)
    # 2. Sorting gives the same order every time
    return sorted(characters)

Watch out

Do not return the set or list(set(text)). The order of a set is not guaranteed, so the numbering would not be repeatable.
Do not strip spaces or line breaks. The model has to be able to write them, so they need IDs.

Check yourself

Why must the vocabulary be built in a repeatable order?

The model learns what each ID means. If the same character got a different ID next time, the trained weights would no longer match the text.

Why is the vocabulary saved with the model?

A different text would produce a different list. The model only works with the numbering it was trained on.