Build a single neuron

Learn it

Words to know

neuron

The basic unit of a network: a weighted sum of its inputs plus a bias, passed through an activation function.

dot product

Multiply two lists of numbers position by position and add up the results. It gives one number.

sigmoid

The function 1 / (1 + e^(-x)). It squeezes any number into the range 0 to 1.

How it works

1. Score: z = x[0] * w[0] + x[1] * w[1] + ... + b. This is the dot product of the inputs and the weights, plus the bias.
2. Output: 1 / (1 + e^(-z)), the sigmoid of the score.
3. A score of 0 gives exactly 0.5. Positive scores give more than 0.5, negative scores less.

Worked example

Input
x = [2, 1]
w = [1, -1]
b = 1

Output
0.880797

A score of 2 becomes 0.88: the neuron leans clearly toward yes. A score of -2 would give 0.12, the same distance on the other side of 0.5.

Why it matters

A layer is many neurons reading the same inputs, each with its own weights. The next two lessons work out how a neuron's weights should change when its answer is wrong.

Your turn

Compute the output of one neuron with a sigmoid activation.

x

the inputs, a list of numbers; may be empty

w

one weight per input

b

the bias

Returns

The sigmoid of dot(x, w) + b. The score stays between -50 and 50, so the plain sigmoid formula is safe.

Input
x = [2, 1]
w = [1, -1]
b = 1

Output
0.880797

Input
x = [1, 2]
w = [2, -1]
b = 0

Output
0.5

Input
x = [0]
w = [7]
b = 0

Output
0.5

Input
x = []
w = []
b = 0

Output
0.5

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Do it in two separate lines: first the score, one number; then the sigmoid of that number.

2. Hint 2: The key step

   The score is the same weighted sum as in the linear model lesson: pair inputs with weights, multiply, add, then add b.

3. Hint 3: The outline

   Set z to b. For each pair from zip(x, w), add value * weight to z. Return 1 / (1 + math.exp(-z)).

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. The score: each input times its weight, plus the bias. Starting the total at b adds the bias once. With no inputs the loop does nothing and the score is the bias.
2. Sigmoid squeezes the score into the range 0 to 1. The score can be any size. The activation turns it into an output that later layers, or a loss, can use.

The same numbered steps are the comments in the solution.

import math


def solve(x, w, b):
    # 1. The score: each input times its weight, plus the bias
    z = b
    for value, weight in zip(x, w):
        z += value * weight
    # 2. Sigmoid squeezes the score into the range 0 to 1
    return 1 / (1 + math.exp(-z))

Watch out

• Apply sigmoid once, to the finished score. Applying it to each input before adding gives a different and wrong result.
• Do not forget the bias. A neuron without one always outputs 0.5 when its inputs are 0.

Check yourself