Neural networks · 5 of 5 · Python

Scale initial weights

Before training, every weight needs a starting value. If all weights start equal, every neuron in a layer computes the same thing and they all learn the same thing, so the layer is no better than one neuron. Random starting values fix that. But their size matters too: too large and the numbers flowing through the network blow up, too small and they fade to nothing. He initialization picks a size that keeps them steady through ReLU layers.

Do the exercise in the app

Learn it

Words to know

fan-in: The number of inputs that feed one neuron.
standard deviation: A measure of how spread out numbers are: the square root of the variance.
ReLU: The function max(0, x). It keeps positive numbers and turns everything else into 0.

How it works

Start from random numbers centred on 0 with a standard deviation of 1, called standard-normal samples. In this exercise they are given to you, so the answer can be checked.
Work out the scale sqrt(2 / fan_in), where fan_in is how many inputs feed each neuron. The 2 makes up for ReLU, which turns about half of the values into 0.
Multiply every sample by the scale. More inputs give a smaller scale, so the sum of many inputs does not grow with their number.

Worked example

Input
samples = [1, -0.5, 2]
fan_in = 8

Output
[0.5, -0.25, 1]

Sample	Scale sqrt(2 / 8)	Weight
1	0.5	0.5
-0.5	0.5	-0.25
2	0.5	1

With 8 inputs every sample is halved. Signs and order stay as they were: scaling only changes the size.

Why it matters

PyTorch layers pick sensible starting weights by a rule like this one. When a deep network refuses to learn from the first step, the starting scale is one of the first things to check.

Your turn

Turn standard-normal samples into He-initialized weights.

samples: a list of random numbers with a standard deviation of 1; may be empty
fan_in: the number of inputs to each neuron; a positive whole number
Returns: Each sample multiplied by sqrt(2 / fan_in), in the same order.

Example 1

Input
samples = [1, -0.5, 2]
fan_in = 8

Output
[0.5, -0.25, 1]

The scale is sqrt(2 / 8) = 0.5, so every sample is halved.

Example 2

Input
samples = [1, -1, 0]
fan_in = 2

Output
[1, -1, 0]

With 2 inputs the scale is sqrt(2 / 2) = 1, and the samples come back unchanged.

Example 3

Input
samples = [1]
fan_in = 8

Output
[0.5]

One sample, halved.

Example 4

Input
samples = []
fan_in = 2

Output
[]

No samples give no weights.

Hints

Each hint gives away a little more. Open one only when you need it.

Hint 1: Where to start
The scale is one number. Work it out once, before touching the samples.
Hint 2: The key step
The scale is math.sqrt(2 / fan_in). Every sample is multiplied by it; nothing is added.
Hint 3: The outline
Set scale to math.sqrt(2 / fan_in). Return a list with sample * scale for each sample.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

The more inputs a neuron has, the smaller its weights start. A neuron adds up fan_in products. Dividing by fan_in under the square root keeps that sum about the same size however many inputs there are.
Scaling changes the spread, not the sign or the order. Multiplying every sample by one number keeps them different from each other and centred on 0. Different starting weights are what let the neurons of a layer learn different things.

The same numbered steps are the comments in the solution.

Python

import math


def solve(samples, fan_in):
    # 1. The more inputs a neuron has, the smaller its weights start
    scale = math.sqrt(2 / fan_in)
    # 2. Scaling changes the spread, not the sign or the order
    return [sample * scale for sample in samples]

Watch out

Multiply, do not add. Adding the scale would shift every weight in the same direction and leave their spread unchanged.
The 2 belongs inside the square root: sqrt(2 / fan_in), not 2 / sqrt(fan_in).

Check yourself

Why can all weights not start at the same value?

Every neuron in a layer would compute the same output and get the same gradient, so they would stay identical for ever.

Why do neurons with more inputs get smaller starting weights?

They add up more terms. Smaller weights keep the total at about the same size, so values neither blow up nor fade from layer to layer.