Encode token positions

Learn it

Words to know

positional encoding

Numbers added to a token's embedding to say where in the sequence the token stands.

embedding

A list of numbers that stands for a token. The model learns these numbers during training.

token

One unit of text as the model sees it: a character, a piece of a word or a whole word.

How it works

1. Each position gets width numbers. They come in pairs: columns 0 and 1 are the first pair, columns 2 and 3 the second, and so on.
2. Pair number i, counted from 0, has its own speed. Its angle at position pos is pos / 10000^(2i / width). The first pair turns fastest; later pairs turn more and more slowly.
3. The even column of a pair holds the sine of that angle, and the odd column holds the cosine.
4. For column j, the pair number is j // 2.

Worked example

Input
length = 3
width = 4

Output
[[0, 1, 0, 1], [0.841471, 0.540302, 0.00999983, 0.99995], [0.909297, -0.416147, 0.0199987, 0.9998]]

The fast pair changes a lot from one position to the next. The slow pair barely moves, so it can tell apart positions that are far from each other.

Why it matters

The tiny GPT in the workshop learns its position numbers instead, with a second embedding table. Either way they are added to the token embeddings before the first attention layer.

Your turn

Build the table of sine and cosine position numbers for a sequence.

length

the number of positions; 0 or more

width

how many numbers each position gets; positive and even

Returns

A list of length rows, each with width numbers. In row pos, column 2i is sin(pos / 10000^(2i / width)) and column 2i + 1 is the cosine of the same angle.

Input
length = 3
width = 4

Output
[[0, 1, 0, 1], [0.841471, 0.540302, 0.00999983, 0.99995], [0.909297, -0.416147, 0.0199987, 0.9998]]

Input
length = 1
width = 4

Output
[[0, 1, 0, 1]]

Input
length = 0
width = 2

Output
[]

Input
length = 2
width = 2

Output
[[0, 1], [0.841471, 0.540302]]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Two loops: over positions, and inside that over the columns of one row.

2. Hint 2: The key step

   Columns j and j + 1 of a pair share one angle. Work out the pair number as j // 2, then the angle, then choose sine or cosine by whether j is even. In Python a power is written **, not ^.

3. Hint 3: The outline

   For each pos and each column j: set angle = pos / 10000 ** (2 * (j // 2) / width). Append math.sin(angle) if j % 2 == 0, otherwise math.cos(angle).

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Columns j and j + 1 of a pair share one angle. j // 2 is 0 for columns 0 and 1, and 1 for columns 2 and 3. A larger pair number divides the position by more, so the angle grows more slowly.
2. Sine in the even column, cosine in the odd one. Sine and cosine of one angle together pin down where the clock hand points. Either one alone would repeat too soon.

The same numbered steps are the comments in the solution.

import math


def solve(length, width):
    table = []
    for pos in range(length):
        row = []
        for j in range(width):
            # 1. Columns j and j + 1 of a pair share one angle
            pair = j // 2
            angle = pos / 10000 ** (2 * pair / width)
            # 2. Sine in the even column, cosine in the odd one
            if j % 2 == 0:
                row.append(math.sin(angle))
            else:
                row.append(math.cos(angle))
        table.append(row)
    return table

Watch out

• Use the pair number j // 2 in the power, not j. With j, the sine and cosine of a pair would get different angles.
• ** binds tighter than / in Python, so pos / 10000 ** p already means pos / (10000 ** p). Brackets the other way round change the result.

Check yourself