AI glossary
Gradient Descent
Gradient descent is an iterative optimization algorithm that finds the minimum of a function by repeatedly adjusting its parameters in the direction opposite to the gradient. In machine learning, the function is the loss, and the parameters are the model’s weights.
Think of it as walking downhill in thick fog. You cannot see the bottom of the valley, so you feel the slope under your feet, take a step in the steepest downward direction, and repeat. This simple intuition drives how modern AI models learn from data.
How gradient descent works
The gradient points in the direction of the steepest increase of a function. By moving against it, you decrease the function value as fast as possible locally. This process relies on the loss function to measure how far the model’s predictions are from the actual targets.
The update rule is straightforward:
theta_new = theta - learning_rate * gradient_of_loss(theta)
The learning rate, or step size, is a hyperparameter set before training begins. It determines how large each step is. You repeat this process until the loss stops decreasing meaningfully or you reach a fixed number of steps.
The learning rate is critical. If it is too small, training takes forever. If it is too large, the model might overshoot the minimum or diverge entirely.
Worked example: minimizing x²
To see this in action, let’s minimize the function f(x) = x². The gradient is f’(x) = 2x, and the minimum is at x = 0.
Start at x = 10 with a learning rate of 0.1: x1 = 10 - 0.1 × 20 = 8 x2 = 6.4 x3 = 5.12
Each step multiplies x by 0.8, so x keeps approaching 0. This convergence is stable and predictable.
Now, change the learning rate to 1.1. Each step multiplies x by -1.2: x1 = -12 x2 = 14.4
The value flips sign and grows larger. The process diverges because the step size is too big for the curvature of the function.
With a learning rate of 0.01, each step multiplies x by 0.98. This converges, but very slowly. This demonstrates why hyperparameter tuning is often necessary to find the right balance.
Here is how this looks in Python:
x = 10.0
lr = 0.1
for step in range(50):
grad = 2 * x # derivative of x**2
x -= lr * grad
print(x) # very close to 0Choosing the learning rate
The learning rate acts as a throttle for how quickly the model learns. A high learning rate speeds up progress but risks instability. A low learning rate ensures stability but can lead to slow convergence or getting stuck in local minima.
In deep learning, you often use learning rate schedules such as warmup, step decay, or cosine decay. These techniques change the learning rate during training to improve final model performance.
Batch, stochastic and mini-batch gradient descent
The main difference between variants is how much data is used to compute the gradient for each update.
Batch gradient descent computes the gradient on the full training set for every update. This is stable but slow on large datasets because it requires processing all data before making a single step.
Stochastic gradient descent (SGD) updates after each single example. It is fast and the noise introduced by single-example updates can help the model escape shallow local minima.
Mini-batch gradient descent updates on small batches, commonly sizes such as 32 to 512 examples. This is the standard in deep learning because it balances the stability of batch methods with the speed of stochastic methods. One pass over the whole training set is an epoch.
Momentum, Adam and other optimizers
Plain gradient descent can struggle with flat regions or noisy gradients. Several optimizers improve on this baseline.
Momentum accumulates a moving average of past gradients. This helps updates keep direction and pass through flat regions faster, reducing oscillation.
RMSProp scales each parameter’s step by a running average of its squared gradients. This adapts the learning rate for each parameter individually, which is useful when features have different scales.
Adam (Kingma and Ba, 2014) combines momentum with per-parameter adaptive learning rates. It is a common default for training neural network models because it handles noise and varying scales well.
AdamW is Adam with decoupled weight decay, which often leads to better generalization.
Common problems
Even with good hyperparameters, gradient descent faces challenges.
A learning rate that is too high causes the loss to oscillate or diverge. A rate that is too low makes training very slow.
Deep networks have non-convex loss surfaces with local minima and saddle points. The algorithm might get stuck in a suboptimal region rather than finding the global minimum.
Vanishing or exploding gradients in deep networks can make learning impossible. If gradients become too small, weights stop updating. If they become too large, weights explode and the model becomes unstable. Read more about vanishing and exploding gradients to understand how to detect and fix these issues.
Features on very different scales create elongated loss surfaces. Gradient descent zig-zags across the valley, slowing down convergence. Standardizing features helps significantly.
Gradient descent and backpropagation
People often confuse these two concepts. They are distinct steps in the training process.
Backpropagation computes the gradient of the loss with respect to every weight in a neural network efficiently. It uses the chain rule to pass error signals backward through the layers.
Gradient descent (or an optimizer like Adam) uses those computed gradients to update the weights. Backpropagation tells you which way is downhill; gradient descent takes the step.
FAQ
What is gradient descent?
Gradient descent is an iterative optimization algorithm that finds the minimum of a function by adjusting parameters in the direction opposite to the gradient. It is the core method used to train neural networks and many other machine learning models.
What is the difference between batch and stochastic gradient descent?
Batch gradient descent uses the entire dataset to compute one gradient update, making it stable but slow. Stochastic gradient descent updates weights after each single example, making it faster but noisier.
What is the Adam optimizer?
Adam is an optimization algorithm that combines momentum with per-parameter adaptive learning rates. It is widely used as a default optimizer for training deep neural networks because it usually works well with little tuning.
Why is the learning rate important?
The learning rate controls the step size during training. If it is too high, the model may diverge. If it is too low, training will be excessively slow. Finding the right value is a key part of hyperparameter tuning.