Python backpropagation is the algorithm that makes neural networks actually learn. We follow the chain rule through a network one layer at a time, work out the gradients by hand, then build a complete training loop from scratch with NumPy. By the end the word “backprop” will feel like something you could rebuild on a napkin.

“Backpropagation is just the chain rule from calculus, applied repeatedly. The genius is not in the math: it is in the software that automates it.”

Yann LeCun, Turing Award Lecture

Last Updated: July 2026 | Tested on: Python 3.14.6, NumPy 2.4.6 | Difficulty: Advanced | Reading Time: 17 minutes

Picture a brand new network. Its weights are random numbers, so its first guesses are basically nonsense. How does it go from nonsense to useful? Through a two step dance that repeats thousands of times. First the forward pass runs the input through the network and produces a prediction, then a loss function measures how wrong that prediction was. Then the backward pass (this is backpropagation) walks that error backward through every layer and figures out how much each weight is to blame for the mistake.

Last, gradient descent nudges every weight a little in the direction that lowers the error. Run that full cycle once over your whole dataset and you have done one epoch. Run it again and again, and the network gets a little smarter each pass.

Here is a quick real life picture. Think of tuning a guitar by ear. You pluck a string (forward pass), you hear that it sounds flat (the loss), and you feel which peg to turn and by how much (the gradient). You give the peg a small twist (the weight update), pluck again, and repeat until it sounds right. Backpropagation is just doing that for millions of “pegs” at once, and the chain rule is the part that tells each peg exactly how much it contributed to the wrong note.

The math tool that makes Python backpropagation possible is the chain rule from calculus. A neuron’s output depends on its input, which depends on the previous layer’s output, which depends on that layer’s input, and so on down the line. The chain rule lets you follow that whole chain and work out how the final loss changes when you wiggle any single weight in any layer. PyTorch handles this for you with automatic differentiation, but knowing what happens under the hood is what lets you fix training when it goes wrong.

Here is what we cover:

• How the chain rule drives backpropagation through layers
• Loss functions: Mean Squared Error (MSE) for regression, cross-entropy for classification
• Gradient descent variants: batch, stochastic, mini-batch
• How learning rate affects convergence
• Building a complete training loop from scratch

Prerequisites

The Python Backpropagation Flow

Read the diagram as one full Python backpropagation loop. The forward pass flows top to bottom: input features go through weighted connections and activation functions until a prediction pops out, and the loss function scores how wrong it was. Then the backward pass flows the other way: using the chain rule, it computes the gradient of the loss for every weight (dL/dw1, dL/dw2, dL/dw3), which is just “how much would the loss change if I nudged this one weight”. Finally the optimizer takes those gradients and updates all the weights at once to make the next prediction a little less wrong.

Loss Functions: Measuring How Wrong You Are

The loss function is just a single number that says how far off the network is. Think of it like a golf score: lower is better, and the whole game is about getting it down. For regression problems (predicting a number, like a house price), MSE takes each error, squares it, and averages them, so big misses hurt a lot more than small ones. For classification (predicting a category, like spam or not spam), Cross-Entropy Loss measures how far the predicted probabilities are from the true answer. Whatever the loss is, backpropagation has exactly one job: push it down.

📄 loss_functions.py: computing MSE and cross-entropy loss

import numpy as np

def mse_loss(y_pred, y_true):
    return np.mean((y_pred - y_true) ** 2)

def cross_entropy_loss(y_pred, y_true):
    epsilon = 1e-15  # Prevent log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Prathamesh tests loss functions
# Regression: predict house price
y_true_reg = np.array([200000, 350000, 150000])
y_pred_good = np.array([195000, 345000, 155000])
y_pred_bad = np.array([100000, 500000, 300000])

print("MSE Loss (Regression):")
print(f"  Good predictions: {mse_loss(y_pred_good, y_true_reg):,.0f}")
print(f"  Bad predictions:  {mse_loss(y_pred_bad, y_true_reg):,.0f}")

# Classification: predict spam (1) or not spam (0)
y_true_cls = np.array([1, 0, 1, 0])
y_pred_conf = np.array([0.95, 0.05, 0.90, 0.10])  # Confident and correct
y_pred_unsure = np.array([0.60, 0.40, 0.55, 0.45])  # Unsure but correct

print(f"\nCross-Entropy Loss (Classification):")
print(f"  Confident correct: {cross_entropy_loss(y_pred_conf, y_true_cls):.4f}")
print(f"  Unsure correct:    {cross_entropy_loss(y_pred_unsure, y_true_cls):.4f}")

▶ Output

MSE Loss (Regression):
  Good predictions: 25,000,000
  Bad predictions:  18,333,333,333

Cross-Entropy Loss (Classification):
  Confident correct: 0.0783
  Unsure correct:    0.5543

What happened here: MSE punishes big errors much harder than small ones, because it squares them. Being off by 150K is not just 3 times worse than being off by 50K, it is 9 times worse (3 squared). That is why the bad predictions score a loss in the billions while the good ones sit at 25 million. Cross-entropy works on probabilities instead: a confident correct guess (0.95 when the true answer is 1) earns almost no loss, while an unsure correct guess (0.60 for the same true 1) still gets a noticeable penalty. So cross-entropy nudges the network to be not just right, but confidently right.

Gradient Descent Visualized

Think of the loss as a hilly valley and the network as a hiker trying to reach the lowest point in the fog. The learning rate is the size of each step. The diagram shows three hikers. A step that is too large makes the hiker leap clean over the valley and bounce up the far side (overshoot, then diverge). A step that is too small means the hiker creeps along and barely moves (very slow convergence).

A well chosen step walks straight down to the bottom. This is exactly why learning rate is the single most important knob in training, and why tricks like learning rate scheduling and warm up exist to change the step size as training goes on.

The Chain Rule in Action

Here is a quick way to feel the chain rule before any symbols. Imagine changing money on a trip: you swap dollars for euros, then euros for rupees. If one dollar buys 0.9 euros, and one euro buys 90 rupees, then one dollar is worth 0.9 * 90 = 81 rupees. You just multiplied the rates along the chain. The chain rule does the same thing for a network: to learn how a weight deep inside affects the final loss, you multiply the “exchange rate” of every layer between them.

The chain rule is the engine inside Python backpropagation. The rule itself is short: if y = f(g(x)), then dy/dx = dy/dg * dg/dx. In plain words, to see how the outside affects the input, you multiply the rates of change along the path. A neural network is just functions stacked inside functions, layer after layer, so to learn how the loss reacts to a weight buried in layer 1, you multiply the gradients of every layer sitting between that weight and the loss.

One nice side effect of seeing it this way: if several of those multiplied numbers are tiny (which is what the sigmoid activation tends to produce), the product shrinks toward zero. That is the famous “vanishing gradient” problem, and now you can see exactly why it happens.

📄 chain_rule_demo.py: manual backpropagation step by step

import numpy as np

# Simple network: 1 input -> 1 hidden -> 1 output
# Aviraj traces gradients by hand

# Forward pass
x = 2.0
w1 = 0.5   # Input to hidden weight
b1 = 0.1   # Hidden bias
w2 = 0.8   # Hidden to output weight
b2 = -0.2  # Output bias
y_true = 1.0

# Layer 1
z1 = x * w1 + b1           # z1 = 2.0 * 0.5 + 0.1 = 1.1
a1 = max(0, z1)             # ReLU: a1 = 1.1 (positive, so pass through)

# Layer 2
z2 = a1 * w2 + b2           # z2 = 1.1 * 0.8 - 0.2 = 0.68
y_pred = z2                  # Linear output for regression

# Loss (MSE for single sample)
loss = (y_pred - y_true) ** 2  # (0.68 - 1.0)^2 = 0.1024

print(f"Forward Pass:")
print(f"  z1 = {x} * {w1} + {b1} = {z1:g}")
print(f"  a1 = ReLU({z1:g}) = {a1:g}")
print(f"  z2 = {a1:g} * {w2} + {b2} = {z2:g}")
print(f"  y_pred = {y_pred:g}")
print(f"  loss = ({y_pred:g} - {y_true})^2 = {loss:.4f}")

# Backward pass (chain rule)
dL_dy = 2 * (y_pred - y_true)    # dLoss/dy_pred
dy_dw2 = a1                      # dy_pred/dw2
dy_da1 = w2                      # dy_pred/da1
da1_dz1 = 1.0 if z1 > 0 else 0  # ReLU gradient
dz1_dw1 = x                      # dz1/dw1

# Chain rule for each weight
dL_dw2 = dL_dy * dy_dw2          # gradient for w2
dL_dw1 = dL_dy * dy_da1 * da1_dz1 * dz1_dw1  # gradient for w1

print(f"\nBackward Pass (gradients):")
print(f"  dL/dy_pred = 2 * ({y_pred:g} - {y_true}) = {dL_dy:.2f}")
print(f"  dL/dw2 = {dL_dy:.2f} * {dy_dw2:g} = {dL_dw2:.3f}")
print(f"  dL/dw1 = {dL_dy:.2f} * {dy_da1} * {da1_dz1} * {dz1_dw1} = {dL_dw1:.3f}")

# Weight update
lr = 0.1
w2_new = w2 - lr * dL_dw2
w1_new = w1 - lr * dL_dw1
print(f"\nWeight Update (lr={lr}):")
print(f"  w2: {w2} -> {w2_new:.4f}")
print(f"  w1: {w1} -> {w1_new:.4f}")

▶ Output

Forward Pass:
  z1 = 2.0 * 0.5 + 0.1 = 1.1
  a1 = ReLU(1.1) = 1.1
  z2 = 1.1 * 0.8 + -0.2 = 0.68
  y_pred = 0.68
  loss = (0.68 - 1.0)^2 = 0.1024

Backward Pass (gradients):
  dL/dy_pred = 2 * (0.68 - 1.0) = -0.64
  dL/dw2 = -0.64 * 1.1 = -0.704
  dL/dw1 = -0.64 * 0.8 * 1.0 * 2.0 = -1.024

Weight Update (lr=0.1):
  w2: 0.8 -> 0.8704
  w1: 0.5 -> 0.6024

What happened here: The prediction (0.68) came out below the target (1.0), so the loss gradient is negative, which is the math telling us “the output is too low, raise it”. Both weight gradients turned out negative too, and the update rule subtracts them (w = w – lr * grad), so subtracting a negative number bumps both weights up. After a single step, w2 moved from 0.8 to 0.87 and w1 from 0.5 to 0.60, both nudging the next prediction closer to 1.0. We just did by hand, for two weights, the exact thing PyTorch’s autograd does automatically for millions of weights at a time.

Complete Training Loop from Scratch

Learning to ride a bicycle works the same way a training loop does. You wobble (a prediction), you notice you are tipping left (the loss), and you steer a touch right to correct (the weight update). One wobble teaches you almost nothing, but a few hundred small corrections turn into balance. Below we put the whole Python backpropagation cycle, forward pass, backward pass, and weight update, into one loop and let it repeat until the network balances. Everything here is plain NumPy, so there is no autograd hiding the work.

📄 training_loop.py: full backpropagation training with NumPy

import numpy as np

class SimpleNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        rng = np.random.default_rng(42)
        self.W1 = rng.normal(0, np.sqrt(2/input_size), (input_size, hidden_size))
        self.b1 = np.zeros(hidden_size)
        self.W2 = rng.normal(0, np.sqrt(2/hidden_size), (hidden_size, output_size))
        self.b2 = np.zeros(output_size)

    def forward(self, X):
        self.z1 = X @ self.W1 + self.b1
        self.a1 = np.maximum(0, self.z1)  # ReLU
        self.z2 = self.a1 @ self.W2 + self.b2
        exp_z = np.exp(self.z2 - np.max(self.z2, axis=1, keepdims=True))
        self.probs = exp_z / exp_z.sum(axis=1, keepdims=True)
        return self.probs

    def backward(self, X, y_onehot, lr=0.01):
        m = X.shape[0]
        # Output gradient
        dz2 = (self.probs - y_onehot) / m
        dW2 = self.a1.T @ dz2
        db2 = dz2.sum(axis=0)
        # Hidden gradient
        da1 = dz2 @ self.W2.T
        dz1 = da1 * (self.z1 > 0)  # ReLU gradient
        dW1 = X.T @ dz1
        db1 = dz1.sum(axis=0)
        # Update weights
        self.W1 -= lr * dW1
        self.b1 -= lr * db1
        self.W2 -= lr * dW2
        self.b2 -= lr * db2

# Vinay trains on a simple classification task
rng = np.random.default_rng(42)
X = rng.normal(0, 1, (200, 4))
y_true = (X[:, 0] + X[:, 1] > 0).astype(int)

y_onehot = np.zeros((200, 2))
y_onehot[np.arange(200), y_true] = 1

net = SimpleNetwork(4, 16, 2)

for epoch in range(201):
    probs = net.forward(X)
    loss = -np.mean(y_onehot * np.log(probs + 1e-15))
    net.backward(X, y_onehot, lr=0.1)

    if epoch % 50 == 0:
        preds = np.argmax(probs, axis=1)
        acc = (preds == y_true).mean()
        print(f"Epoch {epoch:>3d} | Loss: {loss:.4f} | Accuracy: {acc:.1%}")

▶ Output

Epoch   0 | Loss: 0.4374 | Accuracy: 45.0%
Epoch  50 | Loss: 0.0974 | Accuracy: 96.5%
Epoch 100 | Loss: 0.0711 | Accuracy: 97.5%
Epoch 150 | Loss: 0.0582 | Accuracy: 99.0%
Epoch 200 | Loss: 0.0497 | Accuracy: 98.5%

What happened here: The network started out guessing at random (45.0% accuracy at epoch 0, basically a coin flip) and climbed to around 98% accuracy in 200 epochs. Every epoch runs the same three steps you have already seen: a forward pass to make predictions, a backward pass to compute gradients with the chain rule, and a weight update to take one gradient descent step. As the loss slid down from 0.4374 to 0.0497, the network was steadily learning where to draw the decision boundary.

Notice the biggest jump happens early (45% to 96% in the first 50 epochs); after that it is fine tuning. Your exact numbers may wobble a little if you change the random seed, but the shape of the curve will look the same.

Learning Rate: The Most Important Hyperparameter

📄 learning_rate_comparison.py: effect of learning rate on training

import numpy as np

rng = np.random.default_rng(42)
X = rng.normal(0, 1, (200, 4))
y_true = (X[:, 0] + X[:, 1] > 0).astype(int)
y_onehot = np.zeros((200, 2))
y_onehot[np.arange(200), y_true] = 1

# Rahul tests different learning rates
for lr in [0.001, 0.01, 0.1, 1.0, 50.0]:
    net = SimpleNetwork(4, 16, 2)
    final_loss = 0
    for epoch in range(200):
        probs = net.forward(X)
        final_loss = -np.mean(y_onehot * np.log(probs + 1e-15))
        net.backward(X, y_onehot, lr=lr)
    preds = np.argmax(probs, axis=1)
    acc = (preds == y_true).mean()
    print(f"LR={lr:<6} | Final Loss: {final_loss:.4f} | Accuracy: {acc:.1%}")

▶ Output

LR=0.001  | Final Loss: 0.3266 | Accuracy: 66.0%
LR=0.01   | Final Loss: 0.1419 | Accuracy: 94.0%
LR=0.1    | Final Loss: 0.0498 | Accuracy: 98.5%
LR=1.0    | Final Loss: 0.0083 | Accuracy: 100.0%
LR=50.0   | Final Loss: 5.5455 | Accuracy: 53.0%

What happened here: Back to the foggy valley. At a learning rate (LR) of 0.001 the hiker takes baby steps and after 200 epochs has only crawled to 66% accuracy. At 0.01 and 0.1 the steps are well sized and it reaches 94% and 98.5%. On this small, easy problem even LR 1.0 happens to land perfectly (100%), which is a useful reminder that the "best" learning rate depends on the problem, it is not a fixed magic number.

Push it way too far though, to LR 50.0, and the hiker leaps clean over the valley and lands higher up the opposite slope every time: the loss explodes to 5.5 and accuracy collapses back to a coin flip (53%). That is divergence, and it is the failure mode the landscape diagram warned about. Step size is the knob that matters most, so when training will not settle, the learning rate is the first thing to check.

Common Mistakes

Interview Corner

Q: What is the difference between batch, stochastic, and mini-batch gradient descent?

Batch gradient descent looks at the entire dataset before making one weight update. It is stable but slow. Stochastic gradient descent (SGD) updates after every single sample, which is fast but jumpy. Mini-batch gradient descent, the standard in 2026, updates after each batch of 32 to 256 samples, giving you the best trade off between speed and stability. A bonus is that the little bit of noise in mini-batch updates actually helps the optimizer jump out of shallow local minima instead of getting stuck.

Q: How does automatic differentiation work in PyTorch?

PyTorch builds a computational graph during the forward pass, recording every operation. When you call loss.backward(), it traverses this graph in reverse, applying the chain rule at each node to compute gradients. This is called reverse-mode automatic differentiation. It computes gradients for all parameters in a single backward pass, regardless of how many parameters exist. Under the hood it is the same Python backpropagation you traced by hand in this post, just automated.

Practice Exercises

1. Compute the gradients for a 3-layer network by hand on paper, then verify with code.
2. Plot the loss curve for learning rates from 0.0001 to 10, and find the optimal rate.
3. Implement mini-batch gradient descent and compare convergence speed with full-batch.
4. Add gradient clipping to prevent exploding gradients when LR is too high.
5. Compare MSE loss vs cross-entropy loss for a binary classification task.

More in this series:

• DL: Introduction to Neural Networks, Perceptron to Multi-Layer
• PyTorch Dataset and DataLoader: The Real Training Loop
• DL: PyTorch GPU Training, Debugging, and Speedups

Frequently Asked Questions

Interview Questions on Backpropagation

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: Why does the chain rule cause the vanishing gradient problem?

To reach a weight in an early layer, backpropagation multiplies the gradients of every layer in between. If those numbers are all smaller than 1, and sigmoid or tanh activations produce gradients well below 1, the product shrinks toward zero the deeper you go. The early layers then barely update and learning stalls. This is why ReLU (whose gradient is exactly 1 for positive inputs) and residual connections became standard: they keep the multiplied gradients from collapsing.

Q: Your training loss suddenly jumps to NaN after a few epochs. What do you check first?

Start with the learning rate, since it is the most common cause: a rate too high makes the loss overshoot, explode, and then hit NaN, exactly like the LR=50.0 run in this post. Drop it by 10x and see if training settles. If it persists, check for log(0) in the loss (clip probabilities with a small epsilon), un-normalized inputs producing huge activations, and missing gradient clipping. Print the loss and gradient norms each step so you can see where the blow-up begins.

Q: Why do we subtract the gradient during a weight update instead of adding it?

The gradient points in the direction that increases the loss. We want to lower the loss, so we step in the opposite direction, which is why the rule is w = w - lr * grad. In this post's hand-traced example both gradients were negative, so subtracting them actually raised the weights and moved the prediction closer to the target. Adding the gradient would climb the loss surface instead of descending it.

Q: What is the role of the learning rate, and how would you tune it?

The learning rate controls the size of each gradient descent step. Too small and training crawls or stalls in a shallow spot; too large and it overshoots the minimum and can diverge. A common approach is to start around 0.001 to 0.01, watch the loss curve, and adjust: if the loss oscillates or explodes, cut it by 10x; if it drops steadily but far too slowly, raise it. In practice people also use schedules and warm-up to shrink the rate as training progresses.

Q: A teammate applies softmax in the model and also uses PyTorch's CrossEntropyLoss, and the network trains poorly. What is wrong?

PyTorch's CrossEntropyLoss already applies a log-softmax internally, so putting a softmax in the model applies it twice. The doubled squashing flattens the gradients and the network learns slowly or not at all. The fix is to output raw logits from the final layer and let CrossEntropyLoss handle the softmax, or switch to NLLLoss if you truly want softmax in the model.

Q: Why must you call optimizer.zero_grad() inside a PyTorch training loop?

PyTorch accumulates gradients by adding each new backward pass onto whatever is already stored in .grad. If you never clear them, the gradients from previous batches pile up and your updates use a stale, oversized gradient, which corrupts training. Calling optimizer.zero_grad() before each loss.backward() resets them so every step uses only the current batch. Our from-scratch NumPy loop avoids this because it recomputes gradients fresh each pass rather than accumulating.

What’s Next?

You now understand Python backpropagation, the way neural networks actually learn from data: the forward pass makes a prediction, a loss function scores how wrong it is, the chain rule pushes that error backward to find each weight's gradient, and gradient descent nudges every weight down the slope. You also saw why the learning rate is the knob that matters most, and what divergence looks like when it is set too high. That single loop, repeated, is the whole engine behind every model you will train.

Next we swap our from-scratch Python backpropagation code for a production framework. In the PyTorch tutorial, you will build, train, and evaluate a neural network with PyTorch doing the heavy lifting through tensors, autograd, and DataLoaders, so the backprop you just hand-wrote happens for you in one line.

Want the full path from beginner to job-ready? Browse the complete Python + AI/ML tutorial series home for every lesson in order.

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: DL: Activation Functions: sigmoid, tanh, ReLU, softmax

Next: DL: First Neural Network with PyTorch

Series Home: Python + AI/ML Tutorial Series