Machine learning
Machine learning from scratch: 36 lessons
Each lesson teaches one idea in plain words, follows one example by hand, and ends with a small exercise in Python with tests. They go from the math every model uses to a small GPT.
Math foundations
The few pieces of math every later chapter uses: how a model is scored, and how it is nudged to do better.
- Follow a gradientA model starts with a guess and improves it in small steps.
- Sigmoid and ReLU activationsA neuron first adds up its inputs.
- Turn logits into probabilitiesA model that picks between several choices gives each one a raw score.
- Measure classification lossA classifier is a model that picks one of several choices, called classes.
- Predict with a linear modelThe simplest model is a weighted sum.
- Fit a line with gradient descentNow put two earlier ideas together, the linear model and gradient descent, and let the model learn.
Neural networks
Build a network from single neurons by hand, and see how the error travels backwards through it.
- Build a single neuronA neuron is the building block of a neural network, and it does two things you already know.
- Differentiate one neuronA neuron gives an answer, and the answer is off.
- Backpropagate through two layersReal networks have more than one layer, so the error has to travel back through each of them.
- Compose a multilayer perceptronA multilayer perceptron, or MLP, is neurons arranged in layers.
- Scale initial weightsBefore training, every weight needs a starting value.
PyTorch
Do the same math with tensors, the tool real models are written in, and meet three ways to keep numbers at a steady size.
- Reshape and reduce tensorsPyTorch stores numbers in tensors.
- Normalize a feature rowAs numbers pass through many layers, they can drift: some rows of values become huge, others tiny, and training…
- Normalize a training batchBatch normalization solves the same problem as layer normalization, but it looks in the other direction.
- Normalize by root mean squareRMS normalization is a lighter version of layer normalization, and many recent language models use it.
Training
Run a real training loop, and learn to read the signs that it is going well or badly.
- Run an optimizer stepIn chapter 1 you worked out the slopes by hand and updated the numbers yourself.
- Read training signalsWhile a model trains, you watch two numbers.
- Find inactive hidden unitsA unit is another word for a neuron.
- Build a digit classifier headA classifier for handwritten digits takes an image as a list of numbers and gives ten scores, one for each digit from…
NLP
Turn text into numbers a model can use: tokens, vectors, and a sense of word order.
- Look up token embeddingsA model cannot do arithmetic on words, so each token is first given a whole number, its ID.
- Represent text with word countsThe oldest way to turn text into numbers is to count words.
- Classify a sentiment baselineHere is a whole text classifier in a few lines.
- Encode token positionsAn embedding says which token it is, but not where it stands.
Attention & transformers
The step that lets each token look at the tokens before it, and the block that is built around it.
- Compute scaled dot-product attentionAttention lets each token collect information from the other tokens, taking more from the ones that matter to it.
- Combine attention headsOne attention calculation gives each token one way of looking at the others.
- Compose residual attention and normalizationA transformer is the same block stacked many times.
Build GPT
Put it together: a tokenizer, training data, a small GPT, and the loop that makes it write.
- Merge a frequent token pairA GPT does not read letters or whole words.
- Create a reproducible character vocabularyBefore a model can read text, you must decide which tokens exist and give each a number.
- Preserve Unicode and unknown charactersWith a vocabulary in hand, encoding a text is a lookup: each character is replaced by its ID.
- Batch next-token examplesA language model learns one skill: given some text, guess what comes next.
- Split before making windowsTo know whether a model has learned or only memorised, part of the text is kept aside as validation data and never…
- Build a GPT forward passNow the pieces go together.
- Train next-token logitsTraining a language model uses nothing you have not seen.
- Decode autoregressivelyA language model only ever answers one question: what comes next?
- Maintain cached keys and valuesWhile a GPT writes, each new token attends to every token before it.
- Share keys across query headsThe KV cache holds keys and values for every head of every layer, and for a long text that is a great deal of memory.