Sigmoid and ReLU activations

Learn it

Words to know

neuron

The basic unit of a network: a weighted sum of its inputs plus a bias, passed through an activation function.

activation function

A simple function applied to a neuron's sum. It adds a bend, so that layers together can fit curves and not only straight lines.

sigmoid

The function 1 / (1 + e^(-x)). It squeezes any number into the range 0 to 1.

ReLU

The function max(0, x). It keeps positive numbers and turns everything else into 0.

How it works

1. Sigmoid is 1 / (1 + e^(-x)). A large positive x gives a result near 1, a large negative x gives a result near 0, and 0 gives exactly 0.5.
2. For a negative x, the same value can be written e^x / (1 + e^x). Use this form when x is below 0, so the computer never has to raise e to a large positive power.
3. ReLU is max(0, x).

Worked example

Input
xs = [-2, 0, 3]

Output
[[0.119203, 0], [0.5, 0], [0.952574, 3]]

Sigmoid always lands between 0 and 1. ReLU changes nothing above 0 and flattens everything else to 0.

Why it matters

Sigmoid turns a score into a probability in the neuron lesson of chapter 2. ReLU is the activation inside the networks you build after that, including the small GPT.

Your turn

For every number in xs, work out its sigmoid and its ReLU. In Python, e to the power x is written math.exp(x).

xs

a list of numbers, possibly empty; values can be as large as 1000 or as small as -1000

Returns

A list with one pair [sigmoid, relu] for each number, in the same order.

Input
xs = [-2, 0, 3]

Output
[[0.119203, 0], [0.5, 0], [0.952574, 3]]

Input
xs = [0, -1000, 1000]

Output
[[0.5, 0], [0, 0], [1, 1000]]

Input
xs = []

Output
[]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Go through xs one number at a time and build a pair for each. ReLU is one call to max.

2. Hint 2: The key step

   math.exp raises an error when its input is very large. Choose the sigmoid form by the sign of x, so the input to math.exp is never positive.

3. Hint 3: The outline

   For each x: if x >= 0, sigmoid is 1 / (1 + math.exp(-x)); otherwise it is math.exp(x) / (1 + math.exp(x)). ReLU is max(0, x). Append [sigmoid, relu] to the result.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Sigmoid, in the form that keeps the power of e at 0 or below. Both forms give the same value. Choosing by the sign of x means math.exp only ever sees zero or a negative number, which cannot overflow.
2. ReLU keeps a positive number and turns the rest into 0. max(0, x) returns x when it is above 0 and 0 otherwise.

The same numbered steps are the comments in the solution.

import math


def solve(xs):
    result = []
    for x in xs:
        # 1. Sigmoid, in the form that keeps the power of e at 0 or below
        if x >= 0:
            sigmoid = 1 / (1 + math.exp(-x))
        else:
            sigmoid = math.exp(x) / (1 + math.exp(x))
        # 2. ReLU keeps a positive number and turns the rest into 0
        relu = max(0, x)
        result.append([sigmoid, relu])
    return result

Watch out

• Using only 1 / (1 + math.exp(-x)) fails at -1000: Python raises OverflowError, because e^1000 is too large for a float.
• Return the pair in the order asked for: sigmoid first, then ReLU.

Check yourself