Python activation functions are what let a neural network learn curves instead of straight lines, and this hands-on guide covers the five you will meet most in deep learning: sigmoid, tanh, ReLU, Leaky ReLU, and softmax. You will learn what each one does, when to reach for it, why ReLU became the default for hidden layers, and how the vanishing gradient problem quietly killed early deep networks.

“Programming is the art of telling another human being what one wants the computer to do.”

Donald Knuth

Last Updated: July 2026 | Tested on: Python 3.14.6, NumPy 2.4.6, PyTorch 2.12.1 | Difficulty: Intermediate | Reading Time: 16 minutes

You are building a neural network and you hit the first real decision: which activation function goes on each layer? Sigmoid? Tanh? ReLU? Softmax? Pick wrong and the network either trains painfully slowly or never learns at all. This post gives you a way to choose in seconds, with the reasoning behind each option so the choice sticks.

Here is the quick mental picture. An activation function is like the brightness dimmer on a light switch. A plain switch is only on or off, but a dimmer lets each neuron say “a little”, “a lot”, or “not at all”, and that in-between behaviour is what lets a network bend around complex patterns. Without an activation function, stacking layers is pointless: the whole network collapses into one straight line, no matter how deep it is. The activation is the nonlinear twist that makes depth worth having.

Among Python activation functions, the one you pick changes three things at once: how fast the network learns (gradient flow), how much it can learn (representational power), and whether it learns at all (numerical stability). Sigmoid ran the show in the 1990s, tanh improved on it, and then ReLU showed up around 2011 and quietly took over. Knowing why each one wins and where each one breaks is the difference between a network that trains and one that just sits there.

Here is what we cover:

• How sigmoid, tanh, ReLU, Leaky ReLU, and softmax work mathematically
• The vanishing gradient problem and why it kills deep sigmoid networks
• Why ReLU is the default for hidden layers in 2026
• When to use softmax vs sigmoid in the output layer
• How to implement and compare activation functions in PyTorch

Prerequisites

Python Activation Functions at a Glance

This Python activation functions comparison diagram lines up the five common choices, ReLU, Sigmoid, Tanh, Leaky ReLU, and Softmax, with each one’s formula, output range, and best use. ReLU (zero for negatives, pass-through for positives) owns the hidden layers because it dodges the vanishing gradient problem that slows down Sigmoid and Tanh. Softmax sits only on the output layer of multi-class classifiers, where you need probabilities that add up to 1. Picking the right activation per layer is one of the first calls you make when you design a network, and this side-by-side view makes the trade-offs easy to read.

Sigmoid: The Classic S-Curve

Sigmoid was the original activation function, loosely inspired by a biological neuron that fires more as the input grows. It takes any real number and squashes it into the range (0, 1), which makes the output read like a probability. The formula is short: 1 / (1 + e^(-x)). Big positive inputs get pushed toward 1, big negative inputs get pushed toward 0, and anything near zero lands around 0.5. Think of it like a volume knob that can only turn between fully off and fully on, and barely moves once you are near either end.

That “barely moves at the ends” part is the catch: at the extremes the gradient drops to almost nothing, so learning grinds to a halt in deep networks.

📄 sigmoid_demo.py: sigmoid activation and its gradient

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_gradient(x):
    s = sigmoid(x)
    return s * (1 - s)

# Vinay explores sigmoid behavior at different input ranges
inputs = np.array([-10, -5, -1, 0, 1, 5, 10])
print("Sigmoid values and gradients:")
print(f"{'Input':>8} | {'sigmoid(x)':>12} | {'gradient':>12}")
print("-" * 40)
for x in inputs:
    print(f"{x:>8.1f} | {sigmoid(x):>12.6f} | {sigmoid_gradient(x):>12.6f}")

print(f"\nNotice: gradient at x=10 is {sigmoid_gradient(10):.8f}")
print("This is the vanishing gradient problem!")
print("Signal barely flows backward through saturated neurons.")

▶ Output

Sigmoid values and gradients:
   Input |   sigmoid(x) |     gradient
----------------------------------------
   -10.0 |     0.000045 |     0.000045
    -5.0 |     0.006693 |     0.006648
    -1.0 |     0.268941 |     0.196612
     0.0 |     0.500000 |     0.250000
     1.0 |     0.731059 |     0.196612
     5.0 |     0.993307 |     0.006648
    10.0 |     0.999955 |     0.000045

Notice: gradient at x=10 is 0.00004540
This is the vanishing gradient problem!
Signal barely flows backward through saturated neurons.

What happened here: Sigmoid outputs always sit between 0 and 1, but look at the gradient column. It peaks at only 0.25 (right at x=0) and falls to almost nothing for large or small inputs. Now picture a 10-layer network: backpropagation multiplies these gradients layer by layer, and 0.25 to the 10th power is about 0.000001. The learning signal shrinks to a rounding error before it ever reaches the early layers. That is the vanishing gradient problem in one line, and it is exactly why sigmoid is no longer used in hidden layers.

ReLU: The Default Choice

Rectified Linear Unit (ReLU) is almost embarrassingly simple: return zero for negative inputs, return the number unchanged for positive ones. No exponentials, no divisions, no squashing into a tiny range. That simplicity is the whole point. The gradient is either 0 (for negatives) or 1 (for positives), so on the positive side the learning signal flows backward without shrinking at all. Think of a one-way valve: water either passes through at full pressure or it does not pass. Because the positive-side gradient stays at a clean 1, deep ReLU networks train far faster than sigmoid ones. The classic AlexNet result reported roughly 6x faster convergence on its benchmark, and ReLU has been the default for hidden layers ever since.

📄 relu_comparison.py: ReLU vs sigmoid gradient flow

import numpy as np

def relu(x):
    return np.maximum(0, x)

def relu_gradient(x):
    return (x > 0).astype(float)

def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

def leaky_relu_gradient(x, alpha=0.01):
    return np.where(x > 0, 1.0, alpha)

# Rahul compares gradient flow through 10 layers
inputs = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
print("ReLU vs Leaky ReLU:")
print(f"{'Input':>8} | {'ReLU':>8} | {'Grad':>8} | {'LeakyReLU':>10} | {'Grad':>8}")
print("-" * 56)
for x in inputs:
    print(f"{x:>8.1f} | {relu(x):>8.2f} | {relu_gradient(x):>8.2f} | "
          f"{leaky_relu(x):>10.4f} | {leaky_relu_gradient(x):>8.4f}")

# Gradient through 10 layers
layers = 10
relu_signal = 1.0
sigmoid_signal = 1.0
for _ in range(layers):
    relu_signal *= 1.0       # ReLU gradient for positive neurons
    sigmoid_signal *= 0.25   # Sigmoid max gradient

print(f"\nGradient after {layers} layers:")
print(f"  ReLU (positive path):    {relu_signal:.6f}")
print(f"  Sigmoid (best case):     {sigmoid_signal:.10f}")

▶ Output

ReLU vs Leaky ReLU:
   Input |     ReLU |     Grad |  LeakyReLU |     Grad
--------------------------------------------------------
    -2.0 |     0.00 |     0.00 |    -0.0200 |   0.0100
    -0.5 |     0.00 |     0.00 |    -0.0050 |   0.0100
     0.0 |     0.00 |     0.00 |     0.0000 |   0.0100
     0.5 |     0.50 |     1.00 |     0.5000 |   1.0000
     2.0 |     2.00 |     1.00 |     2.0000 |   1.0000

Gradient after 10 layers:
  ReLU (positive path):    1.000000
  Sigmoid (best case):     0.0000009537

What happened here: ReLU keeps the gradient at a full 1 for positive neurons, while sigmoid trims it by 75% at every layer. After 10 layers the ReLU path still carries the whole signal, but the sigmoid path has shrunk to about a millionth of what it started with. There is one trade-off though: ReLU sends the gradient all the way to zero for negative inputs, and a neuron stuck in that zone stops learning. We call that a dead neuron. Leaky ReLU patches the leak by letting a small gradient (0.01) trickle through for negatives, so every neuron stays at least a little bit alive.

Softmax: Multi-Class Probabilities

Softmax does more than squash one number. It turns a whole row of raw scores (called logits) into a set of probabilities that add up to exactly 1, one probability per class. Picture splitting a pizza between classes: every slice is a share of the same pie, and all the slices together make one whole pizza. The exponential in the formula widens the gaps, so the biggest score grabs the largest slice while the small ones get crumbs. This is why softmax always lives on the final layer of a multi-class classifier, where you want to read the output as “how confident am I in each class”.

📄 softmax_demo.py: how softmax converts logits to probabilities

import numpy as np

def softmax(z):
    exp_z = np.exp(z - np.max(z))  # Subtract max for numerical stability
    return exp_z / exp_z.sum()

# Aditi explores softmax behavior
logits = np.array([2.0, 1.0, 0.5])
probs = softmax(logits)
print("Softmax converts logits to probabilities:")
print(f"  Logits:        {logits}")
print(f"  Probabilities: [{probs[0]:.4f}, {probs[1]:.4f}, {probs[2]:.4f}]")
print(f"  Sum:           {probs.sum():.4f}")

# Temperature scaling
for temp in [0.5, 1.0, 2.0, 10.0]:
    scaled = softmax(logits / temp)
    print(f"  Temperature {temp:>4.1f}: [{scaled[0]:.4f}, {scaled[1]:.4f}, {scaled[2]:.4f}]")

▶ Output

Softmax converts logits to probabilities:
  Logits:        [2.  1.  0.5]
  Probabilities: [0.6285, 0.2312, 0.1402]
  Sum:           1.0000
  Temperature  0.5: [0.8438, 0.1142, 0.0420]
  Temperature  1.0: [0.6285, 0.2312, 0.1402]
  Temperature  2.0: [0.4810, 0.2918, 0.2272]
  Temperature 10.0: [0.3616, 0.3272, 0.3112]

What happened here: Softmax turned the logits [2.0, 1.0, 0.5] into the probabilities [0.63, 0.23, 0.14], and notice they add up to 1. The temperature knob controls how confident the result looks. Low temperature (0.5) sharpens the gap and makes the network very sure (about 84% on the top class), while high temperature (10.0) flattens everything toward a near-tie (roughly 36%, 33%, 31%). This is the same temperature setting you tweak when generating text from a large language model: low temperature gives safe, predictable wording, high temperature gives more variety.

Activation Functions in PyTorch

You almost never code Python activation functions by hand in real projects. PyTorch ships them all as ready-made layers, so you just drop one into your model. Think of it like editing the same photo with different filters: the picture going in is identical, but each filter (each activation) reshapes it in its own way. The script below runs the same five inputs through six activations side by side, including two modern ones you will meet in transformers and newer Convolutional Neural Networks (CNNs). Notice how each function reshapes the negative side differently.

📄 pytorch_activations.py: using PyTorch activation functions

import torch
import torch.nn as nn

# Anvi builds models with different activations
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])

activations = {
    "Sigmoid":    nn.Sigmoid(),
    "Tanh":       nn.Tanh(),
    "ReLU":       nn.ReLU(),
    "LeakyReLU":  nn.LeakyReLU(0.01),
    "GELU":       nn.GELU(),    # Used in Transformers
    "SiLU/Swish": nn.SiLU(),    # Used in modern CNNs
}

print(f"{'Input':>7} |", " | ".join(f"{name:>11}" for name in activations))
print("-" * 90)
for val in x:
    outputs = [f"{act(val).item():>11.4f}" for act in activations.values()]
    print(f"{val.item():>7.1f} |", " | ".join(outputs))

print("\nModern recommendations (2026):")
print("  Hidden layers:  ReLU (default), GELU (transformers), SiLU (modern CNNs)")
print("  Binary output:  Sigmoid")
print("  Multi-class:    Softmax")
print("  Hidden (legacy): Tanh (avoid unless specific reason)")

▶ Output

  Input |     Sigmoid |        Tanh |        ReLU |   LeakyReLU |        GELU |  SiLU/Swish
------------------------------------------------------------------------------------------
   -2.0 |      0.1192 |     -0.9640 |      0.0000 |     -0.0200 |     -0.0455 |     -0.2384
   -1.0 |      0.2689 |     -0.7616 |      0.0000 |     -0.0100 |     -0.1587 |     -0.2689
    0.0 |      0.5000 |      0.0000 |      0.0000 |      0.0000 |      0.0000 |      0.0000
    1.0 |      0.7311 |      0.7616 |      1.0000 |      1.0000 |      0.8413 |      0.7311
    2.0 |      0.8808 |      0.9640 |      2.0000 |      2.0000 |      1.9545 |      1.7616

Modern recommendations (2026):
  Hidden layers:  ReLU (default), GELU (transformers), SiLU (modern CNNs)
  Binary output:  Sigmoid
  Multi-class:    Softmax
  Hidden (legacy): Tanh (avoid unless specific reason)

What happened here: Each activation bends the signal in its own way. ReLU is the sharpest of the bunch: zero or pass-through, nothing in between. GELU (the one inside GPT and BERT) and SiLU (Sigmoid Linear Unit), also called Swish (used in EfficientNet), are smooth, rounded cousins of ReLU that treat small negative values more gently instead of snapping them to zero. GELU is the standard pick inside transformer architectures because that smoother curve gives a cleaner gradient, which helps optimization in attention-heavy models that stack many layers.

Quick Reference Table

Common Mistakes

Interview Corner

Q: What is the dying ReLU problem and how do you fix it?

A “dead” ReLU is a neuron that got pushed into the negative zone and got stuck there. Once it sits on the negative side, its gradient is zero, so backpropagation never nudges it back, and it stops learning for good. The usual fixes keep a small slope alive on the negative side: Leaky ReLU (a fixed small slope), PReLU (a slope the network learns), or ELU (a smooth exponential curve for negatives). A more careful (lower) learning rate also helps, since an over-aggressive learning rate is what drives most neurons into the dead zone in the first place. A handful of dead neurons is normal and rarely a problem unless the learning rate is clearly too high.

Q: Why do Transformers use GELU instead of ReLU?

GELU (Gaussian Error Linear Unit) is smoother than ReLU around zero, and that smoothness helps optimization in deep attention-based models where gradients pass through many layers. Where ReLU snaps every negative input to a flat zero, GELU lets small negatives pass through as small nonzero values, so the gradient stays a little cleaner near the cutoff. In practice it tends to give a small but consistent accuracy edge over ReLU on language benchmarks, which is why transformer designs from BERT onward reach for it by default.

Practice Exercises

1. Implement all activation functions and their gradients from scratch with NumPy.
2. Train two identical networks on MNIST, one with sigmoid hidden layers and one with ReLU, then compare how fast each one trains.
3. Create a visualization that shows the gradient magnitudes at each layer for sigmoid vs ReLU in a 10-layer network.
4. Implement GELU from scratch using the formula: x * 0.5 * (1 + erf(x / sqrt(2))).

More in this series:

• Google Colab GPU Setup: Where to Run Deep Learning for Free
• DL: First Neural Network with PyTorch
• PyTorch Dataset and DataLoader: The Real Training Loop

Frequently Asked Questions

Interview Questions on Activation Functions

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: Why does a network need a nonlinear activation function at all? What breaks without one?

Without a nonlinearity, every layer just applies a linear transform, and a stack of linear transforms collapses into a single linear transform. So a 50-layer network with no activations has exactly the same power as one layer: it can only fit straight lines (or flat planes). The nonlinear activation is what lets the network bend and model curved decision boundaries, which is the whole reason depth is useful.

Q: Tanh and sigmoid have the same S-shape, so why is tanh usually preferred over sigmoid in hidden layers?

Tanh is zero-centered: its output ranges from -1 to 1, so activations spread on both sides of zero. Sigmoid outputs only positive values (0 to 1), which makes the gradients of a layer’s weights tend to move in the same direction together and slows convergence. Tanh also has a steeper gradient near zero (up to 1.0 versus sigmoid’s max of 0.25), so a bit more signal survives backprop. Both still saturate at the extremes, which is why ReLU eventually replaced both for hidden layers.

Q: You train a deep network with sigmoid hidden layers and the loss barely drops, while the early layers’ weights hardly change. What is happening and what do you change first?

This is the classic vanishing gradient signature: sigmoid’s gradient maxes out at 0.25, and backprop multiplies those small numbers layer by layer, so by the time the signal reaches the early layers it is near zero and they stop learning. The first fix is to swap the hidden-layer activations to ReLU (or a variant like Leaky ReLU or GELU), whose positive-side gradient is a clean 1. Pairing that with good weight initialization (He initialization) and batch normalization usually gets the early layers training again.

Q: Your multi-class classifier suddenly outputs NaN loss during training, right after a large logit appears. What is the likely cause and the fix?

A raw softmax computes e raised to each logit, and a large logit overflows the float, producing inf and then NaN. The standard fix is the numerically stable softmax: subtract the max logit from every logit before exponentiating, which leaves the result mathematically identical but keeps every exponent at or below zero. In practice you avoid this by letting the framework fuse the two steps, for example PyTorch’s CrossEntropyLoss combines log-softmax and the loss internally for stability.

Q: What is the range and gradient of ReLU, and why does that make it fast to train?

ReLU outputs max(0, x), so its range is 0 to infinity. Its gradient is exactly 1 for positive inputs and 0 for negative ones. Because the positive-side gradient never shrinks below 1, the learning signal passes backward through many layers without decaying, unlike sigmoid or tanh. It is also just a comparison against zero, so it is far cheaper to compute than the exponentials in sigmoid, tanh, or softmax.

Q: When would you use sigmoid on the output layer instead of softmax?

Use sigmoid when the classes are not mutually exclusive. For binary classification you use one sigmoid neuron giving the probability of the positive class. For multi-label problems, where one input can belong to several classes at once (a photo tagged both “beach” and “sunset”), you use one sigmoid per label so each probability is independent. Softmax is only right when exactly one class is correct, because it forces all outputs to compete and sum to 1.

What’s Next?

You now know why Python activation functions matter and which one to reach for on each layer: ReLU (or a smooth cousin like GELU) for hidden layers, sigmoid for a single binary output, and softmax for multi-class output. You also saw the vanishing gradient problem that retired sigmoid from hidden layers, and how the temperature knob on softmax controls confidence. The next big question is how the network actually learns those weights. In the backpropagation tutorial, we build the training algorithm that pushes the error backward through the network and updates the weights to shrink the loss.

Want the full roadmap from Python basics to deep learning? Start at the Python + AI/ML tutorial series home and follow it in order.

Further reading: the official Python documentation is the authoritative source on this.

Related Posts

Previous: DL: Introduction to Neural Networks, Perceptron to Multi-Layer

Next: Backpropagation Explained: Gradient Descent in Python Made Simple

Series Home: Python + AI/ML Tutorial Series