Backpropagation

To improve a neural network with gradient descent you need a gradient for every one of its weights: how much the loss would change if that weight moved a tiny bit. With millions or billions of weights, testing them one by one is hopeless. Backpropagation computes all of them in one sweep. It is the chain rule from calculus, applied systematically from the output back to the input.

It works in two passes. The forward pass runs a batch of examples through the network layer by layer and computes the loss, keeping the intermediate results (the activations) in memory. The backward pass starts at the loss and walks the layers in reverse. Each layer receives the gradient with respect to its output, uses its stored activations to compute the gradients for its own weights, and hands the gradient with respect to its input to the layer before it. The backward pass costs roughly twice as much as the forward pass, however many weights there are.

Nobody writes this by hand anymore. Frameworks like PyTorch and JAX record the operations of the forward pass and differentiate them automatically (automatic differentiation); calling loss.backward() in PyTorch is backpropagation. Those stored activations are a big reason training uses so much more memory than inference, and tricks like activation checkpointing trade extra computation for less memory.

Backpropagation also explains some classic failures. Multiplying many small derivatives through a deep stack of layers can make gradients shrink toward zero (vanishing gradients) or blow up; residual connections, normalization layers and careful initialization exist largely to keep that signal healthy. The method was discovered more than once, and a 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams made it the standard way to train neural networks.