Normalize a feature row

Learn it

Words to know

normalization

Rescaling a group of numbers to a standard size, which keeps training stable.

batch

A group of examples that the model processes together in one step.

variance

A measure of spread: the average of the squared distances from the mean.

epsilon

A tiny number added before dividing or taking a root, so that the calculation is safe when the other part is 0.

How it works

1. For one row, find the mean of its values.
2. Find the variance: the average of the squared distances from the mean. Divide by the number of values N, not by N - 1.
3. Subtract the mean from each value, then divide by sqrt(variance + epsilon).
4. In PyTorch, dim=-1 means the last dimension, the features of each row. keepdim=True keeps the result as a column, so it lines up with the rows when you subtract and divide.

Worked example

Input
rows = [[1, 2, 3], [10, 20, 30]]
epsilon = 0

Output
[[-1.22474, 0, 1.22474], [-1.22474, 0, 1.22474]]

The second row is ten times the first, yet both come out the same. Layer normalization keeps the pattern inside a row and throws away its overall size.

Why it matters

Transformer blocks normalize their values this way, or with the lighter RMS version two lessons on, so a GPT can stack many blocks without its numbers drifting. You use it again in chapter 6.

Your turn

Using PyTorch, apply layer normalization to each row of a table.

rows

a table of numbers; every row has at least one value

epsilon

a number added to the variance, 0 or more, so the division is safe; tiny in practice

Returns

A table of the same shape, as nested Python lists. Each row has its own mean subtracted and is divided by sqrt(variance + epsilon). For the variance, divide the sum of the squared distances by N, not by N - 1.

This lesson needs PyTorch, which does not run in the browser. The app gives you the exercise as a file with its tests, to run on your own computer.

Input
rows = [[1, 2, 3], [10, 20, 30]]
epsilon = 0

Output
[[-1.22474, 0, 1.22474], [-1.22474, 0, 1.22474]]

Input
rows = [[1, 3]]
epsilon = 0

Output
[[-1, 1]]

Input
rows = [[1, 3]]
epsilon = 3

Output
[[-0.5, 0.5]]

Input
rows = [[2, 2]]
epsilon = 0.00001

Output
[[0, 0]]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   You need two numbers per row, a mean and a variance. Ask PyTorch for both along the last dimension.

2. Hint 2: The key step

   Pass keepdim=True to both, so each comes back with shape [rows, 1]. Pass unbiased=False to var so it divides by N.

3. Hint 3: The outline

   Make x from rows as a float64 tensor. Set mean = x.mean(-1, keepdim=True) and variance = x.var(-1, unbiased=False, keepdim=True). Return ((x - mean) / torch.sqrt(variance + epsilon)).tolist().

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. One mean and one variance per row, kept as columns. keepdim=True gives shape [rows, 1]. PyTorch can then subtract that column from every value in the matching row.
2. Centre each row on 0 and scale its spread to 1. Subtracting the mean centres the row. Dividing by the square root of the variance makes its spread 1, and epsilon protects against a variance of 0.

The same numbered steps are the comments in the solution.

import torch


def solve(rows, epsilon):
    x = torch.tensor(rows, dtype=torch.float64)
    # 1. One mean and one variance per row, kept as columns
    mean = x.mean(-1, keepdim=True)
    variance = x.var(-1, unbiased=False, keepdim=True)
    # 2. Centre each row on 0 and scale its spread to 1
    normalized = (x - mean) / torch.sqrt(variance + epsilon)
    return normalized.tolist()

Watch out

• x.var() divides by N - 1 unless you pass unbiased=False. For the row [1, 2, 3] that gives a variance of 1 instead of 0.6667, and every answer is off.
• Without keepdim=True the means have shape [rows]. PyTorch then raises a shape error, or worse, lines them up with the columns when the table is square.
• PyTorch's nn.LayerNorm then multiplies each feature by a learned weight and adds a learned bias. They start at 1 and 0, so a new layer gives the same numbers as this exercise.

Check yourself