Stable token probabilities

Learn it

Words to know

token

The unit a model reads and writes: a word, a piece of a word, or a punctuation mark. Limits and prices are counted in tokens.

logit

A raw score a model gives to one possible next token. It can be any number; higher means more likely.

softmax

The function that turns a list of scores into probabilities that are all between 0 and 1 and add up to 1.

temperature

A setting for how a model picks the next token. Low values make the likeliest token win more often; high values make the output more varied.

How it works

1. Raise e to the power of each score. e is a fixed number, about 2.718; in code this is Math.exp(score) or math.exp(score), not the ^ operator. Every value becomes positive, and a higher score much larger.
2. Add those values up, and divide each one by the total. Now they add up to 1.
3. e to the power 1000 is too large for a decimal number in either language; this is called overflow. So first subtract the largest score from every score. The largest becomes 0, the rest become 0 or negative, and the probabilities come out the same.

Worked example

Input
logits = [1, 2, 3]

Output
[0.0900306, 0.244728, 0.665241]

The three values add up to 1.5032, and the shares add up to 1 (0.9999 as rounded here). Each extra point of score multiplies the share by e, about 2.72.

Why it matters

Every model that writes text ends each step with this. Temperature works by dividing the scores by a number before softmax. A high probability means the model finds a token likely, not that it is true.

Your turn

Turn a list of scores into probabilities with softmax, in the form that cannot overflow.

logits

a list of at least one score; values can be as large as 1000 or as small as -1000

Returns

A list of probabilities in the same order as the scores. They add up to 1.

Input
logits = [1, 2, 3]

Output
[0.0900306, 0.244728, 0.665241]

Input
logits = [0, 0]

Output
[0.5, 0.5]

Input
logits = [1000, 1000]

Output
[0.5, 0.5]

Input
logits = [-1000]

Output
[1]

The checks also try

• two scores one point apart
• one very large score beside a small one
• two very low scores
• scores that are not in order
• the largest score first
• the lowest and the highest score together
• two large scores 100 apart

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Do it in four passes over the list: find the largest score, turn every score into a positive value, add the values up, then divide each value by the total.

2. Hint 2: The key step

   The positive value for a score is e to the power of the score minus the largest score. The largest score itself becomes e to the power 0, which is 1, so nothing can overflow.

3. Hint 3: The outline

   Find largest. Build a list values that holds e to the power score - largest for each score. Add the values up into total. Return a list that holds value / total for each value.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. The largest score, to subtract from every score. After the subtraction the largest power is e to the power 0, which is 1, and every other power is smaller. Nothing can overflow, and at least one value is not 0.
2. A positive value for each score. Raising e to a power always gives a positive number, and it keeps the order: a higher score gives a larger value.
3. The total of the values. Dividing by the total is what makes the results add up to 1.
4. Each value as a share of the total. The shift from step 1 cancels out here: it scales every value and the total by the same amount.

The same numbered steps are the comments in the solution.

function solve(logits) {
  // 1. The largest score, to subtract from every score
  const largest = Math.max(...logits);
  // 2. A positive value for each score
  const values = logits.map((score) => Math.exp(score - largest));
  // 3. The total of the values
  let total = 0;
  for (const value of values) total += value;
  // 4. Each value as a share of the total
  return values.map((value) => value / total);
}

import math


def solve(logits):
    # 1. The largest score, to subtract from every score
    largest = max(logits)
    # 2. A positive value for each score
    values = [math.exp(score - largest) for score in logits]
    # 3. The total of the values
    total = sum(values)
    # 4. Each value as a share of the total
    return [value / total for value in values]

Watch out

• Without the subtraction a score of 1000 overflows, and two scores of -1000 both become 0, which leaves 0 divided by 0.
• Subtract the largest score, not the smallest. With the smallest, a list such as [0, 1000] still raises e to the power 1000.
• In JavaScript: Math.exp(1000) gives Infinity without an error, and Infinity / Infinity is NaN. A failed check that shows NaN usually means the subtraction is missing.
• In Python: math.exp(1000) raises OverflowError: math range error. If you see that message, the subtraction is missing or uses the wrong score.

Check yourself