Backpropagate through two layers

Learn it

Words to know

hidden layer

A layer of neurons between the input and the output. Its values are not seen from outside.

chain rule

If A changes B and B changes C, the effect of A on C is the two effects multiplied together.

ReLU

The function max(0, x). It keeps positive numbers and turns everything else into 0.

preactivation

The value that goes into an activation function: the neuron's weighted sum plus bias, before the bend.

How it works

1. Forward: z = w1 * x, the preactivation; then h = max(0, z), the hidden value; then output = w2 * h. The loss is (output - target)^2 / 2.
2. The error is output - target.
3. Second weight: the output changes by h for each unit of w2. Gradient: error * h.
4. First weight: go back through w2, through the ReLU, and through x. Gradient: error * w2 * x when z is above 0.
5. When z is 0 or below, the ReLU outputs a flat 0. Nothing gets through, and the gradient for w1 is 0.

Worked example

Input
x = 2
w1 = 1
w2 = 3
target = 4

Output
[12, 4]

The output 6 is above the target 4. With a positive input and positive weights, both gradients are positive, so both weights should go down.

Why it matters

The same rule carries the error back through every layer of a deep network. Units whose ReLU blocks the path come back in chapter 4 as dead units.

Your turn

Work out the gradients for both weights of a two-layer model: h = ReLU(w1 * x), output = w2 * h, loss (output - target)^2 / 2.

x

the input, one number

w1

the first-layer weight

w2

the second-layer weight

target

the output we want

Returns

The list [dL/dw1, dL/dw2]. When w1 * x is 0 or below, treat the slope of the ReLU as 0.

Input
x = 2
w1 = 1
w2 = 3
target = 4

Output
[12, 4]

Input
x = 2
w1 = -1
w2 = 3
target = 4

Output
[0, 0]

Input
x = 0
w1 = 1
w2 = 3
target = 4

Output
[0, 0]

Input
x = -2
w1 = -1
w2 = 3
target = 4

Output
[-12, 4]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Run the model forward and keep every value on the way: z, h and the output. The backward pass reads them.

2. Hint 2: The key step

   The gradient for w2 uses h. The gradient for w1 uses w2 and x, and it is 0 unless z is above 0.

3. Hint 3: The outline

   Compute z = w1 * x, h = max(0, z) and error = w2 * h - target. The answer is [error * w2 * x if z > 0 else 0, error * h].

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Forward, keeping the value from before the ReLU. z is the value that went into the ReLU. Above 0 the ReLU lets a gradient through; at 0 or below it blocks it.
2. The second weight is multiplied by h. The output is w2 * h, so each unit of w2 moves the output by h.
3. The first weight is reached back through w2 and the ReLU. This is the chain rule: each link multiplies the error. w2 for the second layer, 1 or 0 for the ReLU, and x for the first layer.

The same numbered steps are the comments in the solution.

def solve(x, w1, w2, target):
    # 1. Forward, keeping the value from before the ReLU
    z = w1 * x
    h = max(0, z)
    error = w2 * h - target
    # 2. The second weight is multiplied by h
    dw2 = error * h
    # 3. The first weight is reached back through w2 and the ReLU
    dw1 = error * w2 * x if z > 0 else 0
    return [dw1, dw2]

Watch out

• Decide the ReLU's slope from z, the value that went into it. The sign of x or of w1 alone does not tell you.
• Do not forget w2 in the first gradient. The first weight can only change the loss by way of the second layer.

Check yourself