Python optimizers are what turn a stack of layers into a model that actually learns, and Adam is the one almost everyone reaches for first, for good reason. This guide shows why Adam is the safe default, when plain SGD (Stochastic Gradient Descent) with momentum can still beat it, and how to choose between SGD, Momentum, RMSprop, Adam, and AdamW using real training runs instead of folklore.

“Adam combines the advantages of AdaGrad and RMSProp into a single update rule.”

Diederik Kingma, Adam paper (ICLR 2015)

Last Updated: July 2026 | Tested on: Python 3.14.6, PyTorch 2.12.1 (Central Processing Unit (CPU)) | Difficulty: Advanced | Reading Time: 11 minutes

Think of training a network like walking downhill in thick fog. Backpropagation tells you which way is downhill (the gradient). The optimizer is the part that actually decides how big a step to take, and in which direction. Plain gradient descent takes the same size step for every weight: it just subtracts lr * gradient and moves on. That sounds fair, but weights in different layers see very different slopes.

Some need tiny careful steps, others need big confident ones. A good optimizer notices this. It takes a custom step size per weight, carries a bit of momentum so it does not stall on flat ground, and shrinks its steps as it gets close to the bottom. That is the whole game, and it is why the adam optimizer usually reaches a good solution faster than vanilla SGD.

Every optimizer on this list was born to fix the one before it. SGD (rooted in the 1950s) was simple but slow. Momentum (1964) added velocity so the path stopped zig-zagging. AdaGrad (2011) gave each parameter its own learning rate. RMSprop (2012, from Hinton) fixed the way AdaGrad let that rate decay to nothing. Adam (2015) bolted momentum and RMSprop together and quietly became the default everyone uses. AdamW (2019) then patched the one thing Adam got wrong about weight decay. So when you pick an optimizer, you are really picking a point on this timeline.

Here is what we cover:

• How each optimizer works mathematically and intuitively
• Why Adam is the default and when SGD beats it
• Learning rate scheduling strategies
• Comparing optimizers on the same task

Read the flowchart top to bottom as a decision guide. Need a safe default? Reach for Adam. Building a Transformer or anything that uses weight decay? Use AdamW. Chasing the very best generalization on a big vision model, and willing to tune? SGD with momentum still wins there. Working with sparse data or NLP (Natural Language Processing) embeddings? Adagrad and RMSprop shine. That is the short version. The rest of this post backs each branch with a real training run so you can see the numbers, not just trust the boxes.

Prerequisites

Python Optimizers Compared

Picture four hikers dropped at the top of the same foggy hill, each told to reach the bottom. One takes cautious, equal-sized steps the whole way (plain SGD). One builds up speed as the slope keeps pointing the same way (SGD with momentum). Two of them size each step to the ground under their own feet (Adam and AdamW). Same hill, same starting point, very different arrival times. The fastest way to feel that difference is to run all four Python optimizers on one problem and read the numbers, so that is exactly what the script below does: same network, same data, same 100 epochs, only the optimizer changes.

📄 optimizer_comparison.py: train the same network with four optimizers

import torch
import torch.nn as nn
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(1000, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_tr = torch.tensor(X_train, dtype=torch.float32)
y_tr = torch.tensor(y_train, dtype=torch.long)
X_te = torch.tensor(X_test, dtype=torch.float32)
y_te = torch.tensor(y_test, dtype=torch.long)

def make_model():
    return nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 2))

# Viraj compares optimizers
optimizers = {
    "SGD (lr=0.01)":       lambda m: torch.optim.SGD(m.parameters(), lr=0.01),
    "SGD+Momentum":        lambda m: torch.optim.SGD(m.parameters(), lr=0.01, momentum=0.9),
    "Adam (lr=0.001)":     lambda m: torch.optim.Adam(m.parameters(), lr=0.001),
    "AdamW (wd=0.01)":     lambda m: torch.optim.AdamW(m.parameters(), lr=0.001, weight_decay=0.01),
}

criterion = nn.CrossEntropyLoss()
for name, opt_fn in optimizers.items():
    torch.manual_seed(42)
    model = make_model()
    opt = opt_fn(model)
    losses = []
    for epoch in range(100):
        model.train()
        loss = criterion(model(X_tr), y_tr)
        opt.zero_grad()
        loss.backward()
        opt.step()
        losses.append(loss.item())

    model.eval()
    with torch.no_grad():
        acc = (model(X_te).argmax(1) == y_te).float().mean()
    print(f"{name:>20}: Final loss={losses[-1]:.4f}, Test acc={acc:.1%}")

▶ Output

       SGD (lr=0.01): Final loss=0.5266, Test acc=80.0%
        SGD+Momentum: Final loss=0.2775, Test acc=87.0%
     Adam (lr=0.001): Final loss=0.2288, Test acc=90.0%
     AdamW (wd=0.01): Final loss=0.2289, Test acc=90.0%

What happened here: Same network, same data, same 100 epochs. The only thing that changed was the optimizer, and look at the spread. Plain SGD limped to 80% because lr=0.01 is just too timid to cover this much ground in 100 epochs. Adding momentum pushed it to 87%, since the velocity term lets it roll through the flat spots instead of crawling. Adam jumped to 90% by giving each weight its own step size: weights with tiny gradients get larger steps, weights with huge gradients get smaller ones, so nothing gets left behind.

AdamW landed at the same 90% here. On a toy 2D problem there is barely anything to over-fit, so its weight decay has little to do. On a real Transformer with millions of parameters, that same weight decay is exactly why AdamW is the go-to optimizer today (at the time of writing). The takeaway: if you only remember one default, make it Adam or AdamW.

Learning Rate Scheduling

📄 lr_scheduler.py: watch StepLR drop the learning rate during training

import torch
import torch.nn as nn

def make_model():
    return nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 2))

torch.manual_seed(42)
model = make_model()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

# Pravin watches how StepLR drops the learning rate over time.
# StepLR multiplies the LR by gamma (0.1 here) every step_size epochs.
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)

print("StepLR schedule (step_size=30, gamma=0.1):")
for epoch in range(100):
    optimizer.step()          # real optimizers update the weights first
    if epoch in (0, 29, 30, 59, 60, 90, 99):
        lr = optimizer.param_groups[0]["lr"]
        print(f"  epoch {epoch:>2}: lr = {lr:.1e}")
    scheduler.step()          # then the scheduler updates the LR

print("\nCommon schedules and when to use them:")
print("  StepLR:          Drop LR by 10x every N epochs (simple, predictable)")
print("  CosineAnnealing: Smooth decay to 0, then restart (good for long training)")
print("  ReduceOnPlateau: Drop LR when val_loss stops improving (adaptive)")
print("  OneCycleLR:      Ramp up then down (best for a limited compute budget)")
print("\nRule of thumb: start with Adam lr=0.001 plus ReduceOnPlateau")

▶ Output

StepLR schedule (step_size=30, gamma=0.1):
  epoch  0: lr = 1.0e-02
  epoch 29: lr = 1.0e-02
  epoch 30: lr = 1.0e-03
  epoch 59: lr = 1.0e-03
  epoch 60: lr = 1.0e-04
  epoch 90: lr = 1.0e-05
  epoch 99: lr = 1.0e-05

Common schedules and when to use them:
  StepLR:          Drop LR by 10x every N epochs (simple, predictable)
  CosineAnnealing: Smooth decay to 0, then restart (good for long training)
  ReduceOnPlateau: Drop LR when val_loss stops improving (adaptive)
  OneCycleLR:      Ramp up then down (best for a limited compute budget)

Rule of thumb: start with Adam lr=0.001 plus ReduceOnPlateau

What happened here: A scheduler changes the learning rate as training goes on. You can see it happen in the output: the learning rate (LR) holds at 1e-02 for 30 epochs, then drops to 1e-03 at epoch 30, then 1e-04 at epoch 60, then 1e-05 at epoch 90. Think of parking a car. You drive in fast across the lot (big steps), then ease off and inch into the actual spot (small steps).

Same idea here: start big to cover ground, shrink the steps near the bottom so you do not overshoot the minimum. StepLR does this on a fixed timer, and schedulers wrap Python optimizers rather than replace them, so you can bolt one onto SGD or AdamW alike. The most practical one in real projects is ReduceOnPlateau, because it does not guess: it watches your validation loss and only cuts the LR when progress actually stalls.

Common Mistakes

Interview Corner

Q: When does SGD outperform Adam?

SGD with momentum often finds flatter minima that generalize better, especially for large-scale image classification (ImageNet). Adam converges faster but can settle in sharper minima. The typical strategy: use Adam for fast prototyping, then switch to SGD+momentum with cosine annealing for final training when you need the last 0.5% accuracy improvement.

Practice Exercises

1. Implement SGD with momentum from scratch (without using PyTorch’s optimizer).
2. Plot training curves for all optimizers on MNIST. Which converges fastest?
3. Try OneCycleLR with max_lr=0.01 and compare with constant LR training.
4. Experiment with Adam epsilon parameter. How does it affect training stability?

More in this series:

• DL: First Neural Network with TensorFlow/Keras
• DL: Transfer Learning with ResNet, VGG Pre-trained Models
• AI: Computer Vision Project, Object Detection with YOLO

Frequently Asked Questions

Interview Questions on Optimizers

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: What is the core difference between SGD, RMSprop, and Adam?

Plain SGD uses one fixed learning rate for every parameter and just subtracts lr * gradient. RMSprop keeps a moving average of squared gradients and divides the gradient by its square root (plus a small epsilon), so each parameter gets its own effective step size (large steps where gradients are tiny, small steps where they are huge). Adam combines both worlds: it adds momentum (a moving average of the gradient itself) on top of RMSprop’s per-parameter scaling. That is why Adam usually converges faster and needs less hand-tuning.

Q: Why prefer AdamW over Adam when you use weight decay?

In plain Adam, L2 regularization gets folded into the gradient and then scaled by the same per-parameter moving averages, so weights with large gradients end up decayed less than intended. AdamW decouples weight decay from the gradient update and applies it directly to the weights, which makes the decay consistent across all parameters. This is why Transformers and most modern architectures default to AdamW.

Q: What do the beta1 and beta2 hyperparameters control in Adam?

beta1 (default 0.9) controls the decay of the first moment, the moving average of the gradient, which acts like momentum. beta2 (default 0.999) controls the decay of the second moment, the moving average of squared gradients, which sets the per-parameter step size. Higher betas mean longer memory and smoother updates. You rarely need to touch them, but lowering beta2 slightly can help when gradients are very noisy.

Q: Scenario: you switch a model from SGD (lr=0.1) to Adam but keep the same learning rate, and the loss immediately explodes to NaN. What is wrong?

The learning rate did not transfer. A healthy SGD rate like 0.1 is far too large for Adam, whose per-parameter scaling already amplifies the effective step. That combination overshoots and diverges to NaN. Drop the Adam learning rate to its normal range around 0.001, and re-tune from there. The rule: never copy a learning rate between optimizers, because each has its own sane range.

Q: Scenario: your training loss keeps bouncing up and down and never settles near the end of a long run. What do you check first?

First suspect a learning rate that stays too high late in training. A constant LR that was fine early on will keep overshooting the minimum once you are close to it, which shows up as that bouncing. Add a scheduler: ReduceOnPlateau to cut the rate when validation loss stalls, or StepLR/CosineAnnealing to shrink it on a timer. Also sanity-check the batch size and confirm the gradients are not being amplified by a missing normalization layer.

Q: What does the epsilon term in Adam do, and when might you change it?

Epsilon (default 1e-8) is a small constant added to the denominator to stop division by zero when the squared-gradient average is near zero. It also caps how large the adaptive step can grow for parameters with tiny gradients. In mixed-precision or very noisy training you sometimes raise it (for example to 1e-4) to improve numerical stability and prevent runaway updates.

What’s Next?

You now have the complete neural network toolkit: architecture, activation functions, backpropagation, regularization, and optimizers. The key takeaway from this post: of the Python optimizers covered here, reach for Adam (or AdamW when weight decay matters) as your safe default at lr=0.001, remember that each optimizer needs its own learning rate range, and add a scheduler like ReduceOnPlateau so the rate shrinks as progress stalls. Keep SGD with momentum in your back pocket for the last bit of generalization on big models. In the CNN tutorial, we apply everything to image data with convolutional neural networks, the architecture that changed computer vision for good.

Want the full learning path from basics to production? Explore the complete Python + AI/ML tutorial series home.

Further reading: the official Python documentation is the authoritative source on this.

Related Posts

Previous: Dropout and Batch Normalization: DL Regularization Explained

Next: DL: CNNs, Convolution, Pooling, Image Classification

Series Home: Python + AI/ML Tutorial Series