Preserve Unicode and unknown characters

Learn it

Words to know

token ID

The whole number that stands for a token.

vocabulary

The full list of tokens a model knows. A token's place in the list gives its ID.

token

One unit of text as the model sees it: a character, a piece of a word or a whole word.

How it works

1. Build a dictionary from character to ID. The first character of vocabulary gets 1, the next 2, and so on. ID 0 is kept for unknown.
2. Go through the text one character at a time and look each one up.
3. A character that is not in the dictionary becomes 0. Uppercase and lowercase are different characters, and spaces are looked up like anything else.

Worked example

Input
text = "hi ho"
vocabulary = [" ", "h", "i"]

Output
[2, 3, 1, 2, 0]

The output has one ID per character, so the text keeps its length. The unknown o still holds its place, as a 0.

Why it matters

Every tokenizer needs a rule for text it has not seen. The workshop's GPT takes a stricter path and refuses a prompt with unfamiliar characters; either way the rule is fixed before training and kept afterwards.

Your turn

Encode a text as a list of token IDs, one per character.

text

any string; may be empty

vocabulary

a list of known characters, each once; the first has ID 1, the second ID 2, and so on

Returns

One ID per character of text, in order. A character that is not in vocabulary gets ID 0.

Input
text = "hi ho"
vocabulary = [" ", "h", "i"]

Output
[2, 3, 1, 2, 0]

Input
text = "a🙂 a"
vocabulary = ["a", " "]

Output
[1, 0, 2, 1]

Input
text = "A"
vocabulary = ["a"]

Output
[0]

Input
text = ""
vocabulary = []

Output
[]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Looking a character up in a list means scanning the list every time. Build a dictionary once and look up there.

2. Hint 2: The key step

   enumerate(vocabulary, start=1) gives each character together with its ID, counting from 1.

3. Hint 3: The outline

   Build ids = {character: i for i, character in enumerate(vocabulary, start=1)}. Return [ids.get(character, 0) for character in text].

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Number the known characters from 1; 0 is kept for unknown. Starting at 1 leaves ID 0 free. If a real character had ID 0, the model could not tell it from an unknown one.
2. Look up each character; an unfamiliar one becomes 0. get with a default never raises an error. A Python string is read one Unicode character at a time. This emoji is one such character; some emoji, such as flags, are several.

The same numbered steps are the comments in the solution.

def solve(text, vocabulary):
    # 1. Number the known characters from 1; 0 is kept for unknown
    ids = {}
    for i, character in enumerate(vocabulary, start=1):
        ids[character] = i
    # 2. Look up each character; an unfamiliar one becomes 0
    return [ids.get(character, 0) for character in text]

Watch out

• enumerate counts from 0 unless you pass start=1. Counting from 0 would give the first known character the unknown ID.
• Do not lowercase or strip the text. The model was trained on the text as it is, and the output should be the same length as the input.

Check yourself