Compose a multilayer perceptron

Learn it

Words to know

hidden layer

A layer of neurons between the input and the output. Its values are not seen from outside.

forward pass

Running input through the model, layer by layer, to get its output.

ReLU

The function max(0, x). It keeps positive numbers and turns everything else into 0.

How it works

1. W holds one row of weights per hidden neuron. For each row, take the dot product with the input x and add that neuron's bias.
2. Pass each result through ReLU. These are the hidden values, one per hidden neuron.
3. Output: the dot product of the hidden values and out_weights, plus out_bias.

Worked example

Input
x = [1, -2]
W = [[1, 0], [0, 1]]
bias = [0, 0]
out_weights = [2, 3]
out_bias = 1

Output
3

The second hidden neuron got -2, and ReLU turned that into 0. Its output weight of 3 then has nothing to multiply.

Why it matters

Every transformer block in a GPT ends with a small MLP like this one. The digit classifier in chapter 4 is one layer of it.

Your turn

Run the forward pass of a small network for one input: a hidden layer with ReLU, then a single output neuron.

x

the input, a list of numbers

W

the hidden weights: one row per hidden neuron, each row as long as x

bias

one bias per hidden neuron

out_weights

one weight per hidden neuron, for the output

out_bias

the bias of the output

Returns

One number: the hidden values dotted with out_weights, plus out_bias.

Input
x = [1, -2]
W = [[1, 0], [0, 1]]
bias = [0, 0]
out_weights = [2, 3]
out_bias = 1

Output
3

Input
x = [0]
W = [[1]]
bias = [0]
out_weights = [2]
out_bias = 0

Output
0

Input
x = [2]
W = [[1]]
bias = [1]
out_weights = [-1]
out_bias = 0.5

Output
-2.5

Input
x = [1, 2]
W = [[1, 1], [-1, 1]]
bias = [-1, 0.5]
out_weights = [2, -1]
out_bias = 0.5

Output
3

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Two stages. First build the list of hidden values, one per row of W. Then combine that list into one number.

2. Hint 2: The key step

   Each hidden value is the linear model from chapter 1, followed by max(0, ...): dot the row with x, add that neuron's bias, apply ReLU.

3. Hint 3: The outline

   For each row and b from zip(W, bias): add up weight * value over zip(row, x), add b, and append max(0, total) to hidden. Then add up h * weight over zip(hidden, out_weights) and add out_bias.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. One hidden value per row of W: weighted sum, bias, ReLU. Every hidden neuron reads the same input x but has its own row of weights and its own bias, so each can react to something different.
2. The output neuron: a weighted sum of the hidden values. The output has no ReLU here, so it can be any number, positive or negative.

The same numbered steps are the comments in the solution.

def solve(x, W, bias, out_weights, out_bias):
    # 1. One hidden value per row of W: weighted sum, bias, ReLU
    hidden = []
    for row, b in zip(W, bias):
        total = b
        for weight, value in zip(row, x):
            total += weight * value
        hidden.append(max(0, total))
    # 2. The output neuron: a weighted sum of the hidden values
    output = out_bias
    for h, weight in zip(hidden, out_weights):
        output += h * weight
    return output

Watch out

• Apply ReLU to each hidden value, not to the final output. This exercise leaves the output as a plain weighted sum.
• Each hidden neuron has its own bias. Pair the rows of W with bias, one to one.

Check yourself