Differentiate one neuron

Learn it

Words to know

backpropagation

Working out the gradient of every weight by going backwards from the loss, one step of the calculation at a time.

chain rule

If A changes B and B changes C, the effect of A on C is the two effects multiplied together.

gradient

The slope of the loss for one number in the model: which way the loss moves, and how fast, when that number goes up a little.

How it works

1. Forward: score z = w * x + b, output p = sigmoid(z), loss L = (p - y)^2 / 2. Halving the square removes the 2 that the slope had in chapter 1.
2. Backward, one link at a time. The loss changes by p - y for each unit of p. The output p changes by p * (1 - p) for each unit of z.
3. Multiply them: delta = (p - y) * p * (1 - p). This is how much the loss changes for each unit of the score.
4. The score changes by x for each unit of w, and by 1 for each unit of b. So the gradients are delta * x and delta.

Worked example

Input
x = 2
w = 0
b = 0
y = 1

Output
[-0.25, -0.125]

Both gradients are negative: raising w or b would lower the loss. That is right, since the output 0.5 is below the target 1 and the input is positive.

Why it matters

A deep network repeats this for every weight in every layer. PyTorch does it when you call loss.backward(), and chapter 4 relies on that.

Your turn

Work out the gradient of the loss for the weight and the bias of one neuron. The neuron computes p = sigmoid(w * x + b) and the loss is (p - y)^2 / 2.

x

the input, one number

w

the weight

b

the bias

y

the target output

Returns

The list [dL/dw, dL/db]: how much the loss changes for each unit of w, and for each unit of b.

Input
x = 2
w = 0
b = 0
y = 1

Output
[-0.25, -0.125]

Input
x = 3
w = 0
b = 0
y = 0

Output
[0.375, 0.125]

Input
x = 1
w = 2
b = -1
y = 0

Output
[0.143735, 0.143735]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Run the neuron forward first. You need the output p before you can go backwards.

2. Hint 2: The key step

   Work out one shared number, delta = (p - y) * p * (1 - p). Both gradients are built from it.

3. Hint 3: The outline

   Compute z = w * x + b and p = 1 / (1 + math.exp(-z)). Set delta to (p - y) * p * (1 - p). Return [delta * x, delta].

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Forward: the neuron's output. The backward pass reuses values from the forward pass, so the output p has to exist first.
2. Backward: how the loss changes with the score. Two links multiplied: p - y from the loss, and p * (1 - p) from the sigmoid. This is the chain rule.
3. The score changes by x per unit of w, and by 1 per unit of b. The last link of the chain. It is why a larger input gives its weight a larger gradient.

The same numbered steps are the comments in the solution.

import math


def solve(x, w, b, y):
    # 1. Forward: the neuron's output
    z = w * x + b
    p = 1 / (1 + math.exp(-z))
    # 2. Backward: how the loss changes with the score
    delta = (p - y) * p * (1 - p)
    # 3. The score changes by x per unit of w, and by 1 per unit of b
    return [delta * x, delta]

Watch out

• Do not leave out p * (1 - p). That factor is the slope of the sigmoid. Without it the answer is the gradient for a different loss, not for this one.
• The order is p - y, not y - p. Swapping them flips the sign, and training would then move away from the target.

Check yourself