Try to run a real AI model and CPU vs GPU vs TPU stops being trivia: it decides whether the model even loads. The marketing is loud and the plain explanation is rare, so this post is the plain explanation. You will learn why AI leans on massively parallel chips, why “does it fit?” matters before “is it fast?”, and exactly what hardware each stage of this series needs.

“People who are really serious about software should make their own hardware.”

Alan Kay

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Beginner | Reading Time: 21 minutes

You do not need any of this hardware to follow the series, and I want to say that upfront so nobody closes the tab thinking AI needs an expensive rig. It does not, not for a long while. But the words CPU, GPU, TPU, and NPU are everywhere in AI, and the errors they cause are the ones that stop beginners cold. So let us build a clear mental model that will outlast every specific chip on sale today. We will run real timing code, do the memory maths that actually decides what you can run, and finish with an honest map of what to buy, rent, or ignore.

Why Matrix Math Loves Being Split Up

Picture a small restaurant run by a chef named Aditi. If Aditi is the only cook and an order comes in for two hundred plates of the same dosa, she can only make them one at a time, no matter how brilliant she is. Now picture the same order handed to a kitchen with two hundred line cooks, each making one plate. The job finishes in the time it takes to make a single dosa. The work did not get smaller, it got shared. That, in one image, is the whole reason AI hardware looks the way it does.

Nearly all of AI, under the hood, is matrix multiplication: huge grids of numbers multiplied together, over and over. And a matrix multiply is made of millions of tiny steps that do not depend on each other. Each output number is just multiply-a-few-pairs-and-add-them-up, and none of those little sums needs to wait for its neighbour. That independence is the golden ticket. Work that can be chopped into pieces that ignore each other is work you can hand to thousands of cooks at once.

Let us prove it with real timing. Below, the same 200 by 200 matrix multiply is done twice: first in plain Python, one operation after another like a single chef, then with NumPy, which quietly hands the job to a parallel, vectorised engine that uses all your cores at once.

📄 matmul_timing.py: one worker vs many, same job

import time, random
import numpy as np

N = 200  # 200 x 200 matrices, so each result cell is a sum over 200 products

random.seed(0)
A = [[random.random() for _ in range(N)] for _ in range(N)]
B = [[random.random() for _ in range(N)] for _ in range(N)]

# 1) Pure Python: one worker doing every multiply in strict order (one chef)
def matmul_python(A, B, n):
    C = [[0.0] * n for _ in range(n)]
    for i in range(n):
        for j in range(n):
            s = 0.0
            for k in range(n):
                s += A[i][k] * B[k][j]
            C[i][j] = s
    return C

t0 = time.perf_counter()
C_py = matmul_python(A, B, N)
t_py = time.perf_counter() - t0

# 2) NumPy: same job handed to a parallel, vectorised engine (many cooks)
An, Bn = np.array(A), np.array(B)
runs = 50
t0 = time.perf_counter()
for _ in range(runs):
    C_np = An @ Bn
t_np = (time.perf_counter() - t0) / runs

same = np.allclose(np.array(C_py), C_np)
print(f"Same result?         {same}")
print(f"Pure Python loops:   {t_py*1000:8.1f} ms")
print(f"NumPy (parallel):    {t_np*1000:8.2f} ms")
print(f"Speed-up:            {t_py / t_np:8.0f}x faster")

▶ Output

Same result?         True
Pure Python loops:      766.4 ms
NumPy (parallel):        0.28 ms
Speed-up:                2783x faster

What happened here: both methods computed the exact same matrix, the first line confirms it. But the plain Python version, doing one multiply at a time, took about three quarters of a second, while NumPy finished the same work in a fraction of a millisecond. The NumPy time is averaged over fifty runs so the number is stable. That gap is not because NumPy is written in C, or not only that.

It is because NumPy spreads the independent little sums across your CPU’s cores and its wide SIMD units, many at once. And here is the punchline: a CPU only has a handful of cores to share the work. A GPU has thousands. Feed this same kind of job to a GPU and the sharing goes much further, which is why the whole AI industry pivoted to graphics chips.

The same code runs on a GPU with almost no change if you use a library like PyTorch, which you will meet later in the series. On a free Google Colab GPU a large multiply that takes half a second on a CPU drops to a few milliseconds. Here is the shape of it, with the CPU figure matching what I measured locally and the GPU figure being an example from a free Colab T4.

📄 torch_matmul.py: the same job, CPU or GPU, decided by one line

import torch, time

# "cuda" means an NVIDIA GPU; fall back to CPU if there is none
device = "cuda" if torch.cuda.is_available() else "cpu"
print("Running on:", device)

x = torch.randn(4096, 4096, device=device)
y = torch.randn(4096, 4096, device=device)

for _ in range(3):        # warm up
    _ = x @ y
if device == "cuda":
    torch.cuda.synchronize()

t0 = time.perf_counter()
for _ in range(10):
    z = x @ y
if device == "cuda":
    torch.cuda.synchronize()
print(f"4096x4096 matmul: {(time.perf_counter() - t0) / 10 * 1000:.0f} ms")

▶ Example output (free Google Colab, T4 GPU)

Running on: cuda
4096x4096 matmul: 9 ms

# For contrast, the identical script on this laptop's CPU printed:
# Running on: cpu
# 4096x4096 matmul: 534 ms

What happened here: one line, the device string, decides where the heavy math runs. On the CPU the big multiply took about half a second, which I measured directly. On the Colab GPU the same code ran in roughly nine milliseconds, near fifty times faster, and the gap only widens as the matrices grow. Note the GPU output is labelled as an example because it needs a GPU I do not have on this machine, but the CPU number above it is real and gives you an honest anchor. This is why training that would take weeks on a CPU finishes in hours on the right GPU.

Inside the Chips: CPU vs GPU vs TPU Architecture

Now let us open the lids. The three chip families you keep hearing about are built on the same trade-off between a few clever workers and many simple ones.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The CPU is the all-rounder in your laptop or phone. It has a few powerful cores, maybe four to sixteen, each very fast and very flexible, able to run your browser, your operating system, and your Python one after another. Think of it as a small team of master chefs: superb at complicated, varied tasks, but only a handful of them. Hand a CPU a job made of a million identical tiny steps and those few chefs still have to grind through them in turns.

The GPU was born to paint pixels on a screen, and a screen is millions of pixels that can all be coloured at once. So a GPU is packed with thousands of small, simple cores that are not clever on their own but are wonderful in a crowd, all doing the same instruction on different data at the same time. That is exactly the shape of a matrix multiply, which is why GPUs became the workhorse of deep learning. Fewer master chefs, a stadium full of line cooks.

The TPU takes one more step. A TPU, and its cousins the NPU and other AI accelerators, is a chip designed for basically one job: matrix multiply. Its heart is a systolic array, a grid of tiny multiply-and-add cells wired directly to each other. Numbers flow through the grid cell to cell, each cell doing its little multiply and passing the result along, without constantly running back to memory in between.

A CPU is a Swiss Army knife, a GPU is a food processor, and a TPU is a machine bolted to the floor that does exactly one thing at enormous speed. That specialisation is why Google builds TPUs for its data centres, and why phones and laptops now ship small NPUs for on-device AI.

VRAM Is the Real Question: Does It Fit?

Here is the thing nobody tells beginners, and it will save you hours of confusion: in the CPU vs GPU vs TPU decision, the first wall you hit is almost never speed. It is memory. A GPU has its own dedicated memory called VRAM, separate from your normal RAM, and the model’s numbers have to physically fit inside it. Think of a fridge: it does not matter how fast a cook named Anvay works if the groceries do not fit in the fridge in the first place. If the model is too big for the VRAM, it does not run slowly, it simply refuses to run.

So how much VRAM does a model need? The rough rule is simple: multiply the number of parameters by the bytes used per number. A parameter stored in fp16 (16-bit) takes 2 bytes. Squeeze it down to 8-bit and it takes 1 byte. Squeeze it to 4-bit and it takes half a byte. That squeezing is called quantization, and you will meet it properly later in the series. For now, watch how much it changes the answer.

📄 vram_fit.py: does this model fit in my GPU?

# How much GPU memory does a model need JUST to sit there (weights only)?
# Rule of thumb: bytes = parameters x bytes-per-number.
# fp16 = 2 bytes, 8-bit = 1 byte, 4-bit = 0.5 bytes per parameter.

BYTES = {"fp16 (16-bit)": 2, "8-bit": 1, "4-bit": 0.5}

models = [
    ("Small (1B params)",   1_000_000_000),
    ("Medium (7B params)",  7_000_000_000),
    ("Large (13B params)", 13_000_000_000),
    ("Huge (70B params)",  70_000_000_000),
]

def gib(num_bytes):
    return num_bytes / (1024 ** 3)   # bytes -> GiB

print(f"{'Model':22} {'fp16':>9} {'8-bit':>9} {'4-bit':>9}")
print("-" * 52)
for name, params in models:
    row = f"{name:22}"
    for label in ("fp16 (16-bit)", "8-bit", "4-bit"):
        need = gib(params * BYTES[label])
        row += f" {need:7.1f}GB"
    print(row)

print()
# A common consumer card has 24 GB. Does a 13B model fit at each precision?
CARD_GB = 24
params = 13_000_000_000
print(f"Fitting a 13B model on a {CARD_GB} GB card (weights only):")
for label, b in BYTES.items():
    need = gib(params * b)
    verdict = "fits" if need < CARD_GB else "does NOT fit"
    print(f"   {label:14} needs {need:5.1f} GB  ->  {verdict}")

▶ Output

Model                       fp16     8-bit     4-bit
----------------------------------------------------
Small (1B params)          1.9GB     0.9GB     0.5GB
Medium (7B params)        13.0GB     6.5GB     3.3GB
Large (13B params)        24.2GB    12.1GB     6.1GB
Huge (70B params)        130.4GB    65.2GB    32.6GB

Fitting a 13B model on a 24 GB card (weights only):
   fp16 (16-bit)  needs  24.2 GB  ->  does NOT fit
   8-bit          needs  12.1 GB  ->  fits
   4-bit          needs   6.1 GB  ->  fits

What happened here: the table turns a scary question into arithmetic. A 7B model needs about 13 GB in fp16, so it just fits on many gaming cards. A 70B model needs a staggering 130 GB in fp16, which is why the biggest models live in data centres across several GPUs. Now look at the bottom block: a 13B model at full fp16 precision needs 24.2 GB and does not fit on a 24 GB card, missing by a whisker.

Quantize it to 8-bit and it drops to 12.1 GB, an easy fit, and 4-bit halves it again. This is why quantization matters so much: it is often the difference between a model you can run at home and one you cannot. Two honest caveats: these numbers are the weights only, and real usage also needs memory for activations and, during training, for gradients, so budget more headroom in practice.

NPUs, Apple Silicon, and Unified Memory

Not all AI runs in a data centre. The laptop or phone in front of you probably has an NPU, a Neural Processing Unit, a small chip tuned for AI that sips power instead of gulping it. That is what runs live captions, photo cleanup, and offline voice typing without draining your battery or sending your data to a server. NPUs are not built to train big models, they are built to run modest ones efficiently, right on the device.

Apple silicon added a twist worth knowing: unified memory. On a normal desktop, the CPU has its RAM and the GPU has its own separate VRAM, and data has to be copied back and forth between the two, like two cooks with two separate pantries passing ingredients across the kitchen. Apple’s chips give the CPU, GPU, and NPU one shared pool of memory instead. So a Mac with 64 GB of unified memory can hand a surprisingly large chunk of that to an AI model, which is why a person named Anvi can run models on a MacBook that would need an expensive dedicated GPU on a Windows or Linux box.

The trade-off is that this shared memory is slower than the specialised VRAM on a high-end GPU, so it is roomy but not the fastest. Roomy often wins for running models at home, since the first question, remember, is whether it fits.

What actually differs between a gaming GPU and a data-centre card like an H100-class accelerator? Less than the price suggests, and more where it counts. Both share the same core idea of thousands of parallel cores. The datacenter card gives you far more VRAM, much faster memory, special high-speed links to gang many cards together, and hardware tuned for the low-precision number formats AI training loves. For learning, a gaming card or free cloud GPU is completely fine. The expensive cards earn their keep only when you train large models at scale.

What Hardware You Actually Need for This Series

Time for the honest, money-saving side of CPU vs GPU vs TPU. Here is what each stage of this series actually asks of your hardware. Notice how far you get on nothing special.

Read that top row again: through the machine learning core of this series, you need no GPU whatsoever. Classic ML, the kind Aviraj uses to predict prices or sort reviews, runs happily on a plain CPU. When you reach deep learning, the free GPUs on Google Colab and Kaggle are genuinely enough to learn on, no purchase required. Renting a cloud GPU by the hour, from a provider like Colab Pro, Lambda, RunPod, or the big clouds, only starts to make sense when a free session times out on you or a model needs more VRAM than the free tier gives.

Buying your own GPU is a want, not a need, and I would wait until you have hit a real limit before spending a rupee on hardware.

CUDA in One Paragraph, and the Error You Will Meet

One word you cannot avoid: CUDA. It is NVIDIA’s software layer that lets your Python talk to an NVIDIA GPU. Libraries like PyTorch are built on top of it, so when a tutorial says “move the model to CUDA,” it just means “put this on the GPU.” The catch is that CUDA is NVIDIA-only, which is a big reason NVIDIA dominates AI: the whole ecosystem grew on their software. Other vendors have their own layers, like AMD’s ROCm, but CUDA is still the default path at the time of writing.

And this is where the VRAM lesson comes back to bite. The single most common error in all of practical AI is CUDA out of memory. It means exactly what the fridge analogy predicted: your model plus its working data asked for more VRAM than the GPU has, so the whole thing stops. You will almost certainly meet it, and there is a later post in this series dedicated to fixing it, using smaller batches, quantization, and other tricks. For now, just know that when you see it, the problem is not that your code is wrong. The problem is that the groceries did not fit in the fridge.

The Chip Landscape as of Mid-2026

Everything above is evergreen: the ideas of parallelism, VRAM, and specialisation do not change. The specific chips do, every single year. So treat this table as a snapshot, not a law. The names and numbers will move, the CPU vs GPU vs TPU categories will not.

If you are reading this a year or two from now, some of those model names will be old news. That is fine. Look past the name to the category, ask how many parallel cores it has and how much memory, and you will always be able to place a new chip on this same map.

Common Mistakes

• Buying a GPU before you need one. Beginners often spend big early, then use it to run code a free Colab notebook would have handled. Learn first, hit a real wall, then buy.
• Confusing RAM with VRAM. Your system RAM and the GPU’s VRAM are separate pools. A model needs to fit in VRAM, and having 64 GB of system RAM does not help if your GPU only has 8 GB.
• Chasing speed before checking fit. People ask which card is fastest when the real question is which card is big enough to hold the model at all. Does it fit comes first.
• Assuming more cores always means faster. A GPU only wins on work that splits into many independent pieces. Run ordinary, step-by-step Python on a GPU and it can be slower than a CPU.
• Reading a CUDA out of memory error as a bug. It is not a code error, it is a capacity error. The fix is a smaller batch, a quantized model, or a bigger GPU, not a hunt through your logic.

Best Practices

• Start on free cloud GPUs. Google Colab and Kaggle give you real GPUs at no cost, ideal for every deep learning exercise in this series.
• Do the VRAM maths before you download. Multiply parameters by bytes per number and check it against your card. It takes ten seconds and saves a failed download.
• Reach for quantization when a model is close. If a model just misses your VRAM in fp16, an 8-bit or 4-bit version often fits with barely any quality loss.
• Write device-agnostic code. The device = "cuda" if torch.cuda.is_available() else "cpu" pattern lets the same script run anywhere, laptop or cloud, unchanged.
• Rent, do not buy, for occasional big jobs. A cloud GPU for a few hours costs less than a coffee run and beats owning hardware that sits idle most of the month.

Conclusion

So the whole CPU vs GPU vs TPU story comes down to a few plain ideas. AI is mostly matrix math, matrix math splits into millions of independent little steps, and chips with many simple cores can share those steps out and finish far faster than a few clever cores working in turn. A CPU is the flexible all-rounder, a GPU is the parallel workhorse, and a TPU or NPU is a chip built for this one job.

The question that decides what you can actually run is not how fast, but does it fit, and that comes down to VRAM and how tightly you quantize the model. Best of all, you need none of this to start: a plain laptop carries you through most of the series, and free cloud GPUs cover the rest.

Next we will get concrete about picking and using that hardware, from setting up a free GPU notebook to knowing when a paid tier is worth it. every lesson in reading order is listed on the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on AI Hardware

How interviewers actually probe this topic: real scenarios, with answers you can say out loud.

Q: Why are GPUs preferred over CPUs for training neural networks?

Neural network training is dominated by matrix multiplication, which decomposes into a huge number of independent multiply-and-add operations. A CPU has only a few cores and processes these largely in sequence, while a GPU has thousands of cores that execute the same operation across many data elements in parallel. Because the work is embarrassingly parallel, the GPU’s design maps onto it almost perfectly, giving order-of-magnitude speed-ups that turn weeks of training into hours.

Q: What is a systolic array and which chip uses it?

A systolic array is a grid of small processing cells, each doing a multiply-and-add, wired directly to its neighbours so data flows cell to cell without constant trips back to memory. It is purpose-built for matrix multiplication. Google’s TPU is the best-known chip built around a systolic array, and the idea also appears in many NPUs. The benefit is very high throughput and energy efficiency for matrix math; the cost is that it is far less flexible than a general GPU or CPU.

Q: How do you estimate the memory needed to load a model, and why does it matter?

As a first approximation, multiply the parameter count by the bytes per parameter for the chosen precision: 2 bytes for fp16, 1 for 8-bit, 0.5 for 4-bit. A 7B model is therefore roughly 13 GB in fp16. It matters because a model that exceeds the GPU’s VRAM will not run at all, producing a CUDA out of memory error. Knowing the estimate lets you pick a precision or a card that fits, and remember to add headroom for activations and, in training, for gradients and optimizer state.

Q: What is CUDA and why is it significant in the AI ecosystem?

CUDA is NVIDIA’s platform for running general computation on its GPUs, and libraries like PyTorch and TensorFlow are built on top of it. Its significance is strategic: because so much of the AI tooling grew on CUDA, it created a strong lock-in that helps explain NVIDIA’s dominance in AI hardware. Competing stacks exist, such as AMD’s ROCm, but at the time of writing CUDA remains the default and best-supported path for GPU compute.

Q: Scenario: a teammate named Anvay says a 70B parameter model will not load on his 24 GB gaming GPU. What do you tell him?

First, confirm the maths: 70B parameters in fp16 need about 130 GB just for the weights, which is far beyond 24 GB, so of course it will not load. Then offer the real options: run a quantized 4-bit version, which drops it to roughly 33 GB and still will not fit alone, so combine it with offloading some layers to system RAM, or pick a smaller model like a 7B or 13B that fits comfortably, or rent a larger cloud GPU or a multi-GPU setup for the full model. The key insight is that this is a capacity constraint, so the answer is about precision, model size, and where the model runs, not about code.

Go deeper: the official Python documentation covers every edge case of this topic.

Related Posts

Previous: AI Training Data: Where Datasets Come From and Why They Matter

Next: How to Choose Hardware for AI: Laptop, Colab, or Cloud GPU

Series Home: Python + AI/ML Tutorial Series