Find inactive hidden units

Learn it

Words to know

neuron

The basic unit of a network: a weighted sum of its inputs plus a bias, passed through an activation function.

preactivation

The value that goes into an activation function: the neuron's weighted sum plus bias, before the bend.

ReLU

The function max(0, x). It keeps positive numbers and turns everything else into 0.

batch

A group of examples that the model processes together in one step.

How it works

1. The table has one row per example and one column per unit. Each number is the value going into that unit's ReLU.
2. Look at one column at a time.
3. If every value in the column is 0 or below, the unit was inactive for the whole batch. Record its column index.
4. A single value above 0 is enough to keep the unit alive.

Worked example

Input
preactivations = [[0, 1, -1], [-2, 2, 0]]

Output
[0, 2]

Exactly 0 counts as inactive: ReLU outputs 0 there, and this course treats its slope as 0 too.

Why it matters

A layer where most units are inactive is wasting its size. A learning rate that is too high and badly scaled starting weights are the usual causes.

Your turn

Find the units that were inactive on every example of a batch.

preactivations

a table with at least one row: one row per example, one column per unit, holding the value that went into each ReLU

Returns

The column indexes whose values are all 0 or below, in increasing order.

Input
preactivations = [[0, 1, -1], [-2, 2, 0]]

Output
[0, 2]

Input
preactivations = [[1, 2]]

Output
[]

Input
preactivations = [[2, -1], [-5, -1]]

Output
[1]

Input
preactivations = [[-1, -1], [-1, 3]]

Output
[0]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   The table is stored row by row, but the question is about columns. Loop over column indexes.

2. Hint 2: The key step

   For column j, look at row[j] in every row. The unit is inactive when all of those are <= 0.

3. Hint 3: The outline

   Set units = len(preactivations[0]). For each j in range(units): if all(row[j] <= 0 for row in preactivations), append j to the result.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Unit j's value in every example. Taking row[j] from each row reads the table downwards, one column at a time.
2. Inactive when none of them is above 0. all stops at the first value above 0. Going through j in order keeps the answer sorted.

The same numbered steps are the comments in the solution.

def solve(preactivations):
    units = len(preactivations[0])
    inactive = []
    for j in range(units):
        # 1. Unit j's value in every example
        column = [row[j] for row in preactivations]
        # 2. Inactive when none of them is above 0
        if all(value <= 0 for value in column):
            inactive.append(j)
    return inactive

Watch out

• Use <= 0, not < 0. A value of exactly 0 gives an output of 0 and no gradient.
• Inactive on one batch is not proof that a unit is dead for good. Another batch might switch it on.

Check yourself