Learning rate

Each step of gradient descent says which way every weight should move to lower the loss function. The learning rate says how far. It is one number, chosen by people before training starts (a hyperparameter, not something the model learns), and it scales every update: w = w - lr * grad.

Getting it right matters more than almost any other setting. Too high, and each step overshoots: the loss bounces around or suddenly shoots up, and training diverges. Too low, and the model learns so slowly that you run out of time or budget, or it settles somewhere mediocre. Useful values are small, roughly between 0.00001 and 0.01 depending on the model and the optimizer, and the reliable way to find a good one is to try a few on short runs and watch the loss.

In practice the learning rate is rarely constant. A schedule changes it over time: a short warmup that ramps it up from near zero, so the first chaotic steps do not wreck the weights, then a slow decay, often shaped like a cosine curve, so the last steps are fine adjustments. Optimizers like Adam also scale the step of each weight on their own, but they still sit on top of one base learning rate.

Fine-tuning usually uses a smaller learning rate than training from scratch. The model already knows a lot, and big steps can erase it; small ones adjust its behavior while keeping most of what it learned. Methods like LoRA, which train far fewer weights, tend to use somewhat higher rates than full fine-tuning.