Measure classification loss

Learn it

Words to know

cross-entropy

The usual loss for choosing between classes: minus the log of the probability the model gave to the right answer.

label

The right answer for one example. For a classifier it is the number of the right class.

loss

One number that says how wrong the model is. Training tries to make it smaller.

batch

A group of examples that the model processes together in one step.

How it works

1. For one example, pick out the probability at the position of the right answer. Call it p.
2. The loss for that example is -log(p), using the natural logarithm. -log(1) is 0, -log(0.5) is about 0.69, and -log(0.1) is about 2.3.
3. log(0) is not defined, so use max(1e-12, p) in place of p.
4. Average the losses of all examples.

Worked example

Input
probabilities = [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]]
targets = [0, 2]

Output
1.32963

The second row put 0.8 on a wrong answer and only 0.1 on the right one. Its loss is more than six times the first row's.

Why it matters

This is the loss a GPT is trained with: at every position, how much probability did the model give to the token that came next?

Your turn

Work out the average cross-entropy loss for a batch of predictions.

probabilities

one row per example; each row is a list of probabilities, one per class

targets

one label per row: a whole number, the position of the right class in that row

Returns

The mean of -log(max(1e-12, p)) over the rows, where p is the probability of the right class. 0 when there are no rows.

Input
probabilities = [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]]
targets = [0, 2]

Output
1.32963

Input
probabilities = [[0.5, 0.5]]
targets = [0]

Output
0.693147

Input
probabilities = []
targets = []

Output
0

Input
probabilities = [[1, 0]]
targets = [1]

Output
27.631

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Only one number in each row matters: the one at index targets[i]. The rest of the row is not used.

2. Hint 2: The key step

   Take -math.log of that number, after raising it to at least 1e-12 with max. Add these up over the rows.

3. Hint 3: The outline

   If there are no rows, return 0. Otherwise go through the rows and targets together with zip, add -math.log(max(1e-12, row[target])) to a running total, and return the total divided by the number of rows.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Nothing to average. Dividing by a count of zero would raise an error, so the empty batch is answered first.
2. The probability given to the right class, kept above zero. log(0) is not defined. The floor of 1e-12 turns it into a large but finite loss.
3. A small probability gives a large loss. log of a number below 1 is negative, so the minus sign makes the loss positive.
4. The average over the examples. Averaging keeps the loss on the same scale whether the batch has 2 rows or 2,000.

The same numbered steps are the comments in the solution.

import math


def solve(probabilities, targets):
    # 1. Nothing to average
    if not probabilities:
        return 0
    total = 0
    for row, target in zip(probabilities, targets):
        # 2. The probability given to the right class, kept above zero
        p = max(1e-12, row[target])
        # 3. A small probability gives a large loss
        total += -math.log(p)
    # 4. The average over the examples
    return total / len(probabilities)

Watch out

• Do not add up the logs of every probability in the row. Only the right class's probability counts.
• Remember the minus sign. log of a probability is zero or negative, and a loss should be zero or positive.

Check yourself