Math foundations · 1 of 6 · Python

Follow a gradient

A model starts with a guess and improves it in small steps. To know which way to step, it measures the slope of its error: does the error go up or down as the number grows? Then it moves the number a little in the downhill direction. Repeat, and the number slides toward the value with the smallest error. This is gradient descent, and it is how every model in this course learns.

Do the exercise in the app

Learn it

Words to know

loss: One number that says how wrong the model is. Training tries to make it smaller.
gradient: The slope of the loss for one number in the model: which way the loss moves, and how fast, when that number goes up a little.
learning rate: How far to move against the gradient on each update. Too small is slow; too large overshoots.

How it works

The loss here is (w - target)^2: the squared distance between the current number w and the value we want. In Python a square is written ** 2, not ^2.
Its slope at w is 2 * (w - target). A negative slope means the loss falls as w grows, so w should go up.
Update: w = w - rate * slope. Subtracting moves w against the slope, which is downhill.
Do that steps times, working out the slope again from the new w each time.

Worked example

Input
w = 0
target = 4
rate = 0.25
steps = 2

Output
3

Step	w	Slope 2(w - 4)	rate × slope	New w
1	0	-8	-2	0 - (-2) = 2
2	2	-4	-1	2 - (-1) = 3

The loss (w - 4)^2 is a bowl with its lowest point at w = 4. From 0, each step moves w downhill, to 2 and then to 3, and the steps shrink as the bowl flattens.

The loss falls from 16 to 4 to 1. The steps also get smaller, 2 and then 1, because the slope flattens as w gets close to 4.

Why it matters

Training a neural network is this same loop with millions of numbers instead of one. Lesson 6 uses it to fit a line, and the PyTorch optimizer in chapter 4 does it for you.

Your turn

Run gradient descent on the loss (w - target)^2 and return where w ends up.

w: the starting value
target: the value with the smallest loss
rate: the learning rate: how much of the slope to move by on each step
steps: how many updates to make; 0 or more
Returns: The value of w after the last update, as a number.

Example 1

Input
w = 0
target = 4
rate = 0.25
steps = 2

Output
3

Two steps take w from 0 to 2 to 3, as in the worked example above.

Example 2

Input
w = 2
target = 8
rate = 0.1
steps = 0

Output
2

With zero steps nothing changes, so the answer is the starting w.

Example 3

Input
w = 9
target = 1
rate = 0.5
steps = 1

Output
1

The slope at 9 is 16. Half of it is 8, and 9 minus 8 lands exactly on the target.

Example 4

Input
w = 0
target = 4
rate = 1
steps = 3

Output
8

A rate of 1 is too large here: w jumps from 0 over the target to 8, back to 0, and to 8 again. It never settles.

Hints

Each hint gives away a little more. Open one only when you need it.

Hint 1: Where to start
You need a loop that runs steps times. Inside it, one number changes: w.
Hint 2: The key step
On each pass, first work out the slope 2 * (w - target) from the current w. Then change w using that slope.
Hint 3: The outline
Repeat steps times: set slope to 2 * (w - target), then set w to w - rate * slope. After the loop, return w.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

The slope of the loss at the current w. The slope is worked out again on every pass, because it changes each time w moves.
Move w against the slope, scaled by the rate. Subtracting a negative slope raises w, and subtracting a positive one lowers it. Either way the loss goes down, if the rate is small enough.
Where w ended up. With zero steps the loop body never runs, and the starting w comes back unchanged.

The same numbered steps are the comments in the solution.

Python

def solve(w, target, rate, steps):
    for _ in range(steps):
        # 1. The slope of the loss at the current w
        slope = 2 * (w - target)
        # 2. Move w against the slope, scaled by the rate
        w = w - rate * slope
    # 3. Where w ended up
    return w

Watch out

Do not add the slope. Adding moves w uphill, and the loss grows on every step.
A rate that is too large overshoots. In this exercise a rate of 1 bounces between two values for ever, and anything above 1 moves further away each step.

Check yourself

Why do we subtract the slope instead of adding it?

The slope points in the direction where the loss grows. Subtracting moves the other way, where the loss shrinks.

What happens when the learning rate is too large?

Each step jumps past the lowest point. The value can bounce from side to side, or move further away every step.