Gradient descent

Training a neural network means finding weights that make its mistakes small. Those mistakes are summed up by a loss function: one number that says how wrong the model is on a batch of examples. Gradient descent is the standard way to lower it. The gradient is the list of partial derivatives of the loss with respect to every weight, and it points in the direction where the loss grows fastest. So you step the opposite way: w = w - lr * grad, for every weight, then repeat, often millions of times.

The learning rate (lr) sets the step size and is the most sensitive knob. Too large and the loss bounces around or blows up; too small and training crawls. It is common to warm it up at the start and lower it as training goes on. Computing the exact gradient over the whole dataset at every step would be far too slow, so stochastic gradient descent (SGD) estimates it from a small random mini-batch. The estimate is noisy, but cheap, and the noise even helps the model avoid getting stuck in poor regions.

Large models rarely use plain SGD. Optimizers like Adam and AdamW keep running averages of past gradients (momentum) and scale each weight's step by how large its gradients have been, which makes training far less fussy. That bookkeeping costs memory: AdamW stores two extra numbers per weight, one reason training needs several times the memory of simply running the model.

A large network's loss landscape is not a neat bowl. It has an enormous number of dimensions, flat plateaus and saddle points, and gradient descent gives no guarantee of finding the lowest point. In practice it reliably finds weights that work well on new data, which is why it trains essentially every modern model. The gradients themselves come from backpropagation; gradient descent decides what to do with them. The idea is old: the mathematician Augustin-Louis Cauchy described it in 1847.