A Python neural network is simpler than it sounds: it starts with the single perceptron that mimics one brain cell and builds up to multi-layer networks that can approximate almost any function. This post explains what neural networks really are, showing how layers of simple math operations stack together to solve problems no single equation could ever handle.

“A neural network is essentially a function approximator. Give it enough neurons and layers, and it can learn any mapping from inputs to outputs.”

Ian Goodfellow, Deep Learning

Last Updated: July 2026 | Tested on: Python 3.14.6, NumPy 2.4.6 | Difficulty: Intermediate | Reading Time: 18 minutes

Part 6 starts here. You have finished 152 posts covering Python fundamentals, intermediate skills, professional tools, data science, and machine learning with scikit-learn. Now you step into deep learning and AI: neural networks, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), transformers, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), agents, and a final capstone project. The next 30 posts take you from understanding a single neuron all the way to building production AI applications.

Every time you ask a voice assistant a question, every time your phone unlocks with your face, every time a streaming service suggests a show you end up binge-watching, a neural network made that call. These systems are not magic. They are layers of simple math operations. Each layer takes numbers in, multiplies them by weights, adds a bias, and passes the result through a function. Stack enough of these layers together and something remarkable happens: the system learns patterns that no human programmer could have written by hand.

The story starts with the perceptron, a model from 1958 that copies a single brain neuron. It takes inputs, multiplies each one by a weight, sums everything up, and then either fires (outputs 1) or stays silent (outputs 0). Think of one neuron like deciding whether to go out for dinner: the weather, your hunger, and your bank balance each get a weight, you add them up, and past a certain point you say yes.

A single perceptron can only solve linearly separable problems, which means it can draw just one straight line to split two classes. The breakthrough came when researchers stacked many perceptrons into layers, creating the multi-layer perceptron (MLP) that can learn curved, twisted, and wildly complex decision boundaries.

Here is what we cover:

• How a single perceptron works: weights, bias, and activation
• Why stacking layers creates a universal function approximator
• Forward pass mechanics from input to prediction
• How to count parameters in a network
• Building your first neural network from scratch with NumPy

Prerequisites

Python Neural Network Architecture

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows the full architecture: three input neurons (one per feature), two hidden layers with ReLU (Rectified Linear Unit) activations, and a two-class output layer with softmax. Every neuron in one layer connects to every neuron in the next, and that is exactly what “fully connected” means. The total parameter count for this network is (3×4 + 4) + (4×3 + 3) + (3×2 + 2) = 16 + 15 + 8 = 39 learnable parameters.

The Perceptron: One Artificial Neuron

The perceptron is the atom of every Python neural network. It takes multiple inputs, multiplies each by a learned weight, adds a bias term, and passes the sum through an activation function. If the result exceeds a threshold, it fires. Think of it as a tiny decision maker: “Given these inputs, should I output yes or no?” The weights determine how important each input is, and the bias shifts the decision boundary.

📄 perceptron.py: a single perceptron from scratch

import numpy as np

def perceptron(inputs, weights, bias):
    """Single perceptron: weighted sum + bias -> step activation."""
    z = np.dot(inputs, weights) + bias
    return 1 if z >= 0 else 0

# AND gate: both inputs must be 1
weights = np.array([0.5, 0.5])
bias = -0.7

# Rahul tests all input combinations
test_cases = [(0, 0), (0, 1), (1, 0), (1, 1)]
print("AND Gate with Perceptron:")
for x1, x2 in test_cases:
    result = perceptron(np.array([x1, x2]), weights, bias)
    print(f"  Input: ({x1}, {x2}) -> Output: {result}")

▶ Output

AND Gate with Perceptron:
  Input: (0, 0) -> Output: 0
  Input: (0, 1) -> Output: 0
  Input: (1, 0) -> Output: 0
  Input: (1, 1) -> Output: 1

What happened here: The perceptron computes 0.5×x1 + 0.5×x2 – 0.7. Only when both inputs are 1 does the sum (0.5 + 0.5 – 0.7 = 0.3) cross zero, so the step function fires and outputs 1. Every other combination stays below zero and outputs 0, which is exactly the AND truth table. We picked the weights and bias by hand here. In a real network, the training algorithm finds them for you through backpropagation.

The XOR Problem: Why One Layer Is Not Enough

A single perceptron can learn AND, OR, and NOT gates, but it fails completely on XOR (exclusive or, which means output 1 when exactly one input is 1). The reason is that XOR is not linearly separable: there is no single straight line you can draw to keep the 1s on one side and the 0s on the other. Picture four points on a sheet of paper where the diagonal pairs share a label. One straight ruler can never split them cleanly. This limitation, proven by Minsky and Papert in 1969, nearly killed neural network research for a decade. The fix is simple: add a hidden layer.

📄 xor_network.py: solving XOR with a hidden layer

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# XOR network: 2 inputs -> 2 hidden neurons -> 1 output
# Niranjan manually sets weights that solve XOR.
# hidden neuron 1 = OR  (fires when at least one input is 1)
# hidden neuron 2 = AND (fires only when both inputs are 1)
# output = OR and NOT AND, which is exactly XOR
W1 = np.array([[20.0, 20.0], [20.0, 20.0]])  # Hidden layer weights
b1 = np.array([-10.0, -30.0])                 # Hidden layer bias (OR, AND)
W2 = np.array([[20.0], [-20.0]])              # Output weights (+OR, -AND)
b2 = np.array([-10.0])                         # Output layer bias

def forward(x):
    hidden = sigmoid(x @ W1 + b1)
    output = sigmoid(hidden @ W2 + b2)
    return output

print("XOR with 2-Layer Network:")
for x1, x2 in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    x = np.array([x1, x2])
    result = forward(x)
    print(f"  Input: ({x1}, {x2}) -> Output: {result[0]:.4f} (rounded: {round(result[0])})")

▶ Output

XOR with 2-Layer Network:
  Input: (0, 0) -> Output: 0.0000 (rounded: 0)
  Input: (0, 1) -> Output: 1.0000 (rounded: 1)
  Input: (1, 0) -> Output: 1.0000 (rounded: 1)
  Input: (1, 1) -> Output: 0.0000 (rounded: 0)

What happened here: The hidden layer turns the inputs into a new representation where XOR finally becomes linearly separable. The first hidden neuron acts like “at least one input is high” (OR), and the second acts like “both inputs are high” (AND). The output neuron then fires when OR is true but AND is false, which is precisely XOR. This is the core insight of deep learning: each layer learns a more useful representation of the data than the one before it.

Building a Multi-Layer Perceptron

Think of an MLP like a team of specialists on an assembly line: the first station spots simple edges, the next combines them into shapes, and the last one recognises the whole object, each layer building on the work of the one before it. A multi-layer perceptron (MLP) generalizes the XOR solution to arbitrary complexity. You stack as many hidden layers as you need, each with as many neurons as you want. The universal approximation theorem guarantees that a network with even a single hidden layer (with enough neurons) can approximate any continuous function to arbitrary precision. In practice, deeper networks with fewer neurons per layer work better than shallow-wide ones.

📄 mlp_from_scratch.py: a complete MLP with NumPy

import numpy as np

class MLP:
    """Multi-Layer Perceptron from scratch using NumPy."""

    def __init__(self, layer_sizes):
        self.weights = []
        self.biases = []
        rng = np.random.default_rng(42)

        for i in range(len(layer_sizes) - 1):
            # He initialization for ReLU
            w = rng.normal(0, np.sqrt(2 / layer_sizes[i]),
                          (layer_sizes[i], layer_sizes[i + 1]))
            b = np.zeros(layer_sizes[i + 1])
            self.weights.append(w)
            self.biases.append(b)

    def relu(self, z):
        return np.maximum(0, z)

    def softmax(self, z):
        exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
        return exp_z / exp_z.sum(axis=1, keepdims=True)

    def forward(self, X):
        self.activations = [X]
        current = X

        for i, (w, b) in enumerate(zip(self.weights, self.biases)):
            z = current @ w + b
            if i < len(self.weights) - 1:
                current = self.relu(z)
            else:
                current = self.softmax(z)
            self.activations.append(current)

        return current

# Viraj creates a network: 4 inputs -> 8 hidden -> 4 hidden -> 3 classes
network = MLP([4, 8, 4, 3])

total_params = sum(
    w.size + b.size for w, b in zip(network.weights, network.biases)
)
print(f"Network architecture: 4 -> 8 -> 4 -> 3")
print(f"Total parameters: {total_params}")
print(f"  Layer 1: {4*8} weights + {8} biases = {4*8+8}")
print(f"  Layer 2: {8*4} weights + {4} biases = {8*4+4}")
print(f"  Layer 3: {4*3} weights + {3} biases = {4*3+3}")

X = np.random.default_rng(42).normal(0, 1, (5, 4))
output = network.forward(X)
print(f"\nSample predictions (5 examples, 3 classes):")
for i, probs in enumerate(output):
    predicted = np.argmax(probs)
    print(f"  Example {i+1}: [{probs[0]:.3f}, {probs[1]:.3f}, {probs[2]:.3f}] -> Class {predicted}")

▶ Output

Network architecture: 4 -> 8 -> 4 -> 3
Total parameters: 91
  Layer 1: 32 weights + 8 biases = 40
  Layer 2: 32 weights + 4 biases = 36
  Layer 3: 12 weights + 3 biases = 15

Sample predictions (5 examples, 3 classes):
  Example 1: [0.188, 0.533, 0.279] -> Class 1
  Example 2: [0.343, 0.224, 0.433] -> Class 2
  Example 3: [0.196, 0.518, 0.285] -> Class 1
  Example 4: [0.333, 0.322, 0.345] -> Class 2
  Example 5: [0.324, 0.320, 0.356] -> Class 2

What happened here: The network has 91 learnable parameters across three layers (40 + 36 + 15). He initialization (scaling by sqrt(2/n)) keeps activations from vanishing or exploding in ReLU networks. The forward pass pushes the inputs through each layer, using ReLU for the hidden layers and softmax for the output. The predicted probabilities are all over the place and even a little confident in spots (0.533 for one example) only because the random starting weights happen to lean that way. The network has learned nothing yet, so these guesses mean nothing. Training via backpropagation (next post) is what makes the network actually learn.

The Forward Pass Step by Step

The forward pass is like passing a form through an office: each desk reads what the last desk wrote, stamps it with its own rule, and hands it to the next. Let us trace one sample through a tiny Python neural network by hand so you can see exactly what each desk does to the numbers.

📄 forward_pass_trace.py: tracing every calculation

import numpy as np

rng = np.random.default_rng(42)
W1 = rng.normal(0, 0.5, (2, 3))
b1 = np.zeros(3)
W2 = rng.normal(0, 0.5, (3, 2))
b2 = np.zeros(2)

# Pravin traces forward pass for one sample
x = np.array([1.5, -0.5])
print(f"Input: {x}")

z1 = x @ W1 + b1
print(f"\nLayer 1 pre-activation (z = x*W + b):")
print(f"  z1 = {z1}")

a1 = np.maximum(0, z1)
print(f"  ReLU(z1) = {a1}")

z2 = a1 @ W2 + b2
print(f"\nLayer 2 pre-activation:")
print(f"  z2 = {z2}")

exp_z2 = np.exp(z2 - np.max(z2))
softmax_out = exp_z2 / exp_z2.sum()
print(f"  Softmax = {softmax_out}")
print(f"  Sum of probabilities: {softmax_out.sum():.4f}")
print(f"  Predicted class: {np.argmax(softmax_out)}")

▶ Output

Input: [ 1.5 -0.5]

Layer 1 pre-activation (z = x*W + b):
  z1 = [-0.00660337 -0.29222928  0.88838327]
  ReLU(z1) = [0.         0.         0.88838327]

Layer 2 pre-activation:
  z2 = [0.39062123 0.34548867]
  Softmax = [0.51128122 0.48871878]
  Sum of probabilities: 1.0000
  Predicted class: 0

What happened here: The forward pass has exactly two kinds of operations: linear (multiply by weights, add bias) and nonlinear (the activation function). ReLU zeroed out the first two hidden neurons because their pre-activations were negative, leaving only the third alive. That is how a network creates sparse activations, where many neurons sit silent for any given input. The softmax then turned the two raw logits into probabilities that add up to 1, and the slightly larger one (class 0) wins.

Parameter Counting: How Big Is Your Network?

Counting parameters is like counting the knobs on a giant mixing board: each weight and bias in a Python neural network is a knob the training process can turn. More knobs means more flexibility, but also more ways to get lost. Every connection between two neurons is one weight, and every neuron adds one bias, so the total tells you how much the network has to learn.

📄 param_counting.py: counting parameters for common architectures

def count_params(layers):
    """Count total parameters in a fully connected network."""
    total = 0
    details = []
    for i in range(len(layers) - 1):
        weights = layers[i] * layers[i + 1]
        biases = layers[i + 1]
        layer_total = weights + biases
        details.append(f"  Layer {i+1}: {layers[i]}x{layers[i+1]} + {layers[i+1]} = {layer_total:,}")
        total += layer_total
    return total, details

# Aditi compares different architectures
architectures = {
    "Simple":   [784, 128, 10],
    "Medium":   [784, 256, 128, 10],
    "Deep":     [784, 512, 256, 128, 64, 10],
    "Wide":     [784, 1024, 10],
}

for i, (name, layers) in enumerate(architectures.items()):
    total, details = count_params(layers)
    if i > 0:
        print()
    print(f"{name}: {' -> '.join(str(l) for l in layers)}")
    for d in details:
        print(d)
    print(f"  Total: {total:,} parameters")

▶ Output

Simple: 784 -> 128 -> 10
  Layer 1: 784x128 + 128 = 100,480
  Layer 2: 128x10 + 10 = 1,290
  Total: 101,770 parameters

Medium: 784 -> 256 -> 128 -> 10
  Layer 1: 784x256 + 256 = 200,960
  Layer 2: 256x128 + 128 = 32,896
  Layer 3: 128x10 + 10 = 1,290
  Total: 235,146 parameters

Deep: 784 -> 512 -> 256 -> 128 -> 64 -> 10
  Layer 1: 784x512 + 512 = 401,920
  Layer 2: 512x256 + 256 = 131,328
  Layer 3: 256x128 + 128 = 32,896
  Layer 4: 128x64 + 64 = 8,256
  Layer 5: 64x10 + 10 = 650
  Total: 575,050 parameters

Wide: 784 -> 1024 -> 10
  Layer 1: 784x1024 + 1024 = 803,840
  Layer 2: 1024x10 + 10 = 10,250
  Total: 814,090 parameters

What happened here: The first layer always dominates the parameter count because the input size (784 pixels for MNIST) is large. The Wide network has more parameters than the Deep one despite having fewer layers, because a single wide layer with 1024 neurons creates a massive weight matrix. Modern deep learning favors deeper networks over wider ones.

How Neural Networks Transform Data

Imagine two colours of thread tangled into a knot. You cannot separate them with one straight cut, but if you stretch and fold the fabric just right, the two colours line up and a single snip does the job. That folding is exactly what each layer does to your data, reshaping it until the classes fall apart cleanly.

📄 data_transformation.py: watching data flow through layers

import numpy as np

rng = np.random.default_rng(42)
n = 100
theta = np.linspace(0, 4 * np.pi, n)
r = np.linspace(0.5, 2, n)

x0 = np.column_stack([r * np.cos(theta), r * np.sin(theta)])
x1 = np.column_stack([r * np.cos(theta + np.pi), r * np.sin(theta + np.pi)])

X = np.vstack([x0, x1])
y = np.array([0] * n + [1] * n)

W1 = rng.normal(0, 0.5, (2, 8))
b1 = np.zeros(8)
W2 = rng.normal(0, 0.5, (8, 4))
b2 = np.zeros(4)

# Anvi watches the transformation
z1 = X @ W1 + b1
a1 = np.maximum(0, z1)
z2 = a1 @ W2 + b2
a2 = np.maximum(0, z2)

print(f"Input shape: {X.shape} (200 samples, 2 features)")
print(f"After layer 1: {a1.shape} (200 samples, 8 features)")
print(f"After layer 2: {a2.shape} (200 samples, 4 features)")
print(f"\nSparsity after ReLU:")
print(f"  Layer 1: {(a1 == 0).mean():.1%} of activations are zero")
print(f"  Layer 2: {(a2 == 0).mean():.1%} of activations are zero")
print(f"\nKey insight: Each layer transforms the data into a new")
print(f"representation where the classes are easier to separate.")

▶ Output

Input shape: (200, 2) (200 samples, 2 features)
After layer 1: (200, 8) (200 samples, 8 features)
After layer 2: (200, 4) (200 samples, 4 features)

Sparsity after ReLU:
  Layer 1: 50.0% of activations are zero
  Layer 2: 43.0% of activations are zero

Key insight: Each layer transforms the data into a new
representation where the classes are easier to separate.

What happened here: The input data lives in 2D where the two spirals are intertwined. Layer 1 projects it into 8 dimensions, and ReLU zeroes out about half the values (sparse activation). Layer 2 further transforms it into 4 dimensions. Even with random weights, the network changes the data geometry. With trained weights, these transformations would unravel the spirals into a representation where a simple linear classifier can separate them.

Common Mistakes

Interview Corner

Q: Why do we need nonlinear activation functions?

Without activation functions, a multi-layer network collapses into a single linear transformation: y = W3(W2(W1x)) = Wx. No matter how many layers you stack, the network can only model linear relationships. Nonlinear activation functions let each layer warp the data space, enabling the network to learn curved decision boundaries and complex patterns.

Q: What is the universal approximation theorem?

It states that a feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of R^n to any desired accuracy. The catch: it does not tell you how many neurons you need, and deeper networks learn more efficiently because they build hierarchical features.

Practice Exercises

1. Modify the perceptron to implement an OR gate. What weights and bias do you need?
2. Build a 3-layer MLP (input: 10, hidden: 20, hidden: 10, output: 5) and count the total parameters.
3. Implement a forward pass using both ReLU and sigmoid activations. Compare the output ranges.
4. Create a function that counts how many neurons are “dead” (always zero after ReLU) for random inputs.
5. Research: Why did Minsky and Papert’s 1969 book nearly kill neural network research?

More in this series:

• Machine Learning Interview Questions: 40-Question Checkpoint
• Backpropagation Explained: Gradient Descent in Python Made Simple
• DL: First Neural Network with PyTorch

Frequently Asked Questions

Interview Questions on Neural Networks

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: What is the difference between a perceptron and a multi-layer perceptron?

A perceptron is a single artificial neuron: it takes inputs, computes a weighted sum plus a bias, and applies an activation. It can only separate data with one straight line, so it fails on problems like XOR. A multi-layer perceptron stacks neurons into hidden layers, and with nonlinear activations between them it can carve out curved, complex decision boundaries and approximate almost any function.

Q: Why is XOR impossible for a single-layer perceptron?

XOR is not linearly separable, meaning no single straight line can keep all the 1s on one side and all the 0s on the other. A single perceptron can only draw one linear boundary, so it can never model XOR. Adding a hidden layer lets the network transform the inputs into a new representation where the classes finally become linearly separable.

Q: How do you count the parameters in a fully connected layer?

For a layer that maps n inputs to m outputs, the weight matrix has n times m entries and there are m biases, so the layer has (n times m) + m learnable parameters. For example, a 784 to 128 layer has 784 times 128 plus 128, which is 100,480 parameters. Summing this across all layers gives the total size of the network.

Q: Why do hidden layers need a nonlinear activation function?

Without a nonlinearity, stacking linear layers just produces another linear function: W3(W2(W1x)) collapses to a single matrix Wx. That means extra layers add no power at all. A nonlinear activation like ReLU lets each layer bend the data space, which is what enables the network to learn curved boundaries and hierarchical features.

Q: Your deep network trains but the loss barely moves and early-layer weights hardly change. What do you check first?

This is the classic vanishing gradient symptom. First check the activation functions: sigmoid or tanh in deep hidden layers squash gradients toward zero, so switch hidden layers to ReLU. Next check weight initialization, using He initialization for ReLU so activations neither vanish nor explode. Also verify the learning rate is not so small that updates are negligible, and confirm the data is scaled sensibly.

Q: You build an MLP with 10 million parameters to classify 500 labeled samples. Training accuracy hits 100 percent but test accuracy is poor. What went wrong?

The network is massively overparameterized for the dataset, so it memorized the 500 samples instead of learning a general pattern, which is overfitting. Shrink the network to match the data size, add regularization such as dropout or weight decay, and gather more training data if possible. A validation set will reveal the gap between memorization and generalization early.

Q: What does the softmax function do in the output layer?

Softmax turns a vector of raw scores (logits) into a probability distribution: every value lands between 0 and 1, and they all sum to 1. This lets you read each output as the network’s confidence in a class, and the argmax gives the predicted class. It is the standard choice for multi-class classification output layers.

What’s Next?

You now understand Python neural network architecture, from single perceptrons all the way to multi-layer networks. You saw why one layer cannot solve XOR, how a hidden layer fixes it, how the forward pass turns inputs into predictions, and how to count the parameters your network has to learn. But notice that we handpicked the weights for our XOR network. So how does a network learn the right weights from data on its own? In the activation functions tutorial, we look at how different activation functions affect learning, and in the backpropagation tutorial, we build the training algorithm that makes neural networks actually learn.

Want the full roadmap from Python basics to production AI? Explore the complete Python + AI/ML tutorial series home.

Reference: the complete, always-current details live in the official Python documentation.

Related Posts

Previous: Google Colab Graphics Processing Unit (GPU) Setup: Where to Run Deep Learning for Free

Next: DL: Activation Functions: sigmoid, tanh, ReLU, softmax

Series Home: Python + AI/ML Tutorial Series