Renting tokens from an API is easy; the bill and the data both leave the building. LLM serving is the craft of running an open Large Language Model yourself, fast enough for real users, and this post walks both ends of it with tested Python: Ollama on your laptop for development, vLLM in production, plus the KV-cache math that decides how many users one Graphics Processing Unit (GPU) can actually hold.

“The cheapest, fastest, and most reliable components are those that aren’t there.”

Gordon Bell

Last Updated: July 2026 | Tested on: Python 3.14.6 (standard library only) | Difficulty: Advanced | Reading Time: 19 minutes

Serving is its own discipline, separate from training and separate from prompt work. Think of it like a restaurant kitchen: the recipe (the model) matters, but whether dinner service runs smoothly comes down to the stove, the pass, and how many plates you can push out per hour. Two tools own the two ends of this job right now. Ollama is the easy stove on your counter for cooking one dish at a time while you develop. vLLM is the commercial kitchen that feeds hundreds of tables at once.

The mechanics under both, the KV cache and batching, are what actually decide your throughput, so we teach those first and treat the tools as the current best way to reach them.

Ollama: An Open Model on Your Laptop

The best part of local LLM serving is that you stop paying per token and stop sending data to anyone. Ollama makes this a two-command affair: it downloads an open model and starts a small server, and that server speaks the same OpenAI-compatible dialect your app already knows. So if you built the chatbot from the earlier project, you do not rewrite anything. You point its OpenAI client at Ollama and keep going. First, the two shell commands that pull a small model and confirm the server is up.

🖥️ terminal: pull a small open model and start serving

# Download a small open model (3B params, a few GB). One-time.
ollama pull llama3.2:3b

# Ollama runs a local server on port 11434 and exposes an
# OpenAI-compatible endpoint at http://localhost:11434/v1
ollama serve   # (already running as a background service on most installs)

Now the one-line swap. In the chatbot project, the OpenAI adapter created its client with OpenAI(), which reads a cloud Application Programming Interface (API) key. To talk to Ollama instead, you give that same client a base_url pointing at the local server and any non-empty api_key string, because the local server does not check it. Nothing else in the app changes. Here is the swap, then a script that parses a real response captured from the local server so the extraction logic runs with no server needed.

📄 the one-line change: same client, local base_url

from openai import OpenAI

# Cloud (before):
#   client = OpenAI()   # reads OPENAI_API_KEY, calls api.openai.com

# Local with Ollama (after): point at the local server, key is ignored.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "In one line, what is paneer paratha?"}],
)
print(resp.choices[0].message.content)

📄 ollama_call.py: parse a real OpenAI-compatible response body

import json

# This is a real response body captured from a local Ollama server's
# OpenAI-compatible endpoint (POST http://localhost:11434/v1/chat/completions).
# We parse it here so the extraction logic runs with no server needed.
captured = r'''
{
  "id": "chatcmpl-476",
  "object": "chat.completion",
  "created": 1751558400,
  "model": "llama3.2:3b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Paneer paratha is a whole-wheat flatbread stuffed with spiced, crumbled paneer."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 26, "completion_tokens": 47, "total_tokens": 73}
}
'''

resp = json.loads(captured)
reply = resp["choices"][0]["message"]["content"]
usage = resp["usage"]
print("model :", resp["model"])
print("reply :", reply[:70], "...")
print("tokens: {} in + {} out = {} total".format(
    usage["prompt_tokens"], usage["completion_tokens"], usage["total_tokens"]))
print("finish:", resp["choices"][0]["finish_reason"])

▶ Output

model : llama3.2:3b
reply : Paneer paratha is a whole-wheat flatbread stuffed with spiced, crumble ...
tokens: 26 in + 47 out = 73 total
finish: stop

What happened here: the response shape is byte-for-byte the same as the cloud API, which is the whole point. The choices[0].message.content holds the reply, usage reports the token counts, and finish_reason tells you it stopped cleanly instead of hitting a length cap. Because the envelope matches, your parsing code, your streaming loop, your token logging, all of it keeps working when you switch between a local model and a cloud one. Ollama is perfect for building and testing offline, demos on a plane, and anything privacy-sensitive. What it is not built for is a hundred users hitting it at once, and that is exactly where the next two sections take us.

Quantization in Practice: 4-bit vs 8-bit vs fp16

Before a model can serve anyone, its weights have to fit in memory, and that is where quantization earns its keep. A weight is just a number, and you get to choose how many bytes you spend storing each one. Full serving precision (fp16) spends 2 bytes per weight. Quantize to 8-bit and you spend 1 byte; quantize to 4-bit and you spend half a byte. The trade is a small drop in quality for a large drop in memory.

Think of it like saving a photo as a slightly smaller JPEG: from across the room you cannot tell, and the file is a quarter of the size. The two formats you will meet by name are GGUF (the format Ollama and llama.cpp use, great for Central Processing Unit (CPU) and mixed setups) and AWQ (a GPU-focused 4-bit scheme common with vLLM). Here is the memory math for an 8B model at each precision.

📄 quant_vram.py: weights memory at each precision

# Weights memory = (parameters) x (bytes per parameter).
# Bytes per parameter depends on the precision you load the model in.
PARAMS = 8_000_000_000  # an 8B open model, e.g. Llama 3.1 8B

PRECISION = {              # bytes per weight
    "fp16 (16-bit)":  2.0,   # "full" serving precision, no quantization
    "int8 (8-bit)":   1.0,   # AWQ / GPTQ / bitsandbytes 8-bit
    "int4 (4-bit)":   0.5,   # GGUF Q4 / AWQ 4-bit, the laptop-friendly one
}
# Rough quality signal vs the fp16 baseline, from published benchmarks.
QUALITY_VS_FP16 = {"fp16 (16-bit)": 100.0, "int8 (8-bit)": 99.5, "int4 (4-bit)": 98.0}

print(f"Weights-only VRAM for an {PARAMS/1e9:.0f}B model:\n")
print(f"{'precision':<16}{'GB for weights':>16}{'quality vs fp16':>18}")
for name, bytes_per in PRECISION.items():
    gb = PARAMS * bytes_per / (1024 ** 3)
    print(f"{name:<16}{gb:>13.1f} GB{QUALITY_VS_FP16[name]:>16.1f}%")

fp16_gb = PARAMS * 2.0 / (1024 ** 3)
int4_gb = PARAMS * 0.5 / (1024 ** 3)
print(f"\n4-bit uses {fp16_gb/int4_gb:.0f}x less weight memory than fp16.")
print("Note: this is weights only. The KV cache adds more per active request.")

▶ Output

Weights-only VRAM for an 8B model:

precision         GB for weights   quality vs fp16
fp16 (16-bit)           14.9 GB           100.0%
int8 (8-bit)             7.5 GB            99.5%
int4 (4-bit)             3.7 GB            98.0%

4-bit uses 4x less weight memory than fp16.
Note: this is weights only. The KV cache adds more per active request.

What happened here: the same 8B model needs 15 GB at fp16 but only 3.7 GB at 4-bit, which is the difference between “needs a data-center GPU” and “runs on a gaming laptop.” The quality numbers are a rough signal, not a promise, since the real hit depends on the model and the task, but the pattern holds: 8-bit is almost free quality-wise, and 4-bit costs a couple of points for a huge memory win.

That is why local tools default to 4-bit GGUF and why production teams on big GPUs often stay at fp16 or 8-bit for full quality. One honest caveat: these are weights only. Every active request also needs its own KV cache, and that number, not the weights, is usually what caps how many users you can serve. We compute it two sections down.

Knowledge Distillation

Quantization shrinks a model you already have, but there is a second way to make a big model small, and it produces a genuinely smaller model rather than a compressed copy of the old one. It is called knowledge distillation. You start with a large, accurate model, the teacher, and run it across a big dataset. For every example you record not just the teacher’s final answer but its full set of soft output probabilities, the confidence it spread across every possible label. Then you train a much smaller model, the student, to reproduce those probability distributions instead of only matching the plain right-or-wrong labels. A good human teacher does the same thing: they do not only tell you the correct answer, they tell you which wrong answers were tempting and which were nonsense, and that extra shading is where a lot of the real understanding lives. Those soft targets carry the teacher’s nuance, so the student learns far more than the hard labels alone could teach it and keeps most of the accuracy at a fraction of the size and cost.

It helps to line up the three ways teams make models smaller, because they are easy to mix up. Quantization keeps the same model and the same number of weights but stores each weight in fewer bits, trading a little precision for a lot of memory. Pruning also keeps the same model, but snips away the individual weights or connections that contribute almost nothing, leaving a leaner version of the original. Distillation is the odd one out: it does not touch the big model’s weights at all, it trains a brand-new, smaller architecture from scratch to imitate the big one’s behavior. Quantization and pruning compress what already exists; distillation grows a fresh, compact model that learned from a large one.

The best-known example, and the one this series keeps pointing back to, is DistilBERT. It was distilled from BERT, Google’s landmark language model, and the numbers are the reason the technique caught on: DistilBERT is roughly 40% smaller and about 60% faster than BERT while keeping about 97% of its performance on standard language tasks. That is a model you can run on cheaper hardware, ship to a phone, or serve to far more users per GPU, all for a few points of quality you will rarely notice. Whenever you meet a Distil-something model, that Distil prefix is the tell that a smaller student learned its craft from a larger teacher.

vLLM: PagedAttention and Continuous Batching

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

When you need to serve many users at once on a GPU, vLLM is the unambiguous production standard at the time of writing (mid-2026), with SGLang as its main rival for high-concurrency workloads. Its reputation rests on two ideas shown in the diagram above, and both are about one scarce resource: the KV cache. The KV cache is the memory that stores what the model has already read so it does not recompute every token each step. It grows with every active request, and it is the thing that runs out first.

PagedAttention is the first idea. Instead of reserving one big contiguous block of memory per request (most of which sits empty because you do not know how long the answer will be), vLLM splits the KV cache into small fixed-size pages and hands them out on demand, the way an operating system pages RAM. Almost no memory is wasted, so more requests fit at once. Continuous batching is the second idea.

Old-style static batching runs a fixed group together and makes everyone wait for the slowest member before starting the next group. vLLM instead swaps a finished request out and a waiting one in on the very next step, so the batch never has idle slots. The simulation below shows why that matters, using nothing but the standard library.

📄 batching_sim.py: static batching vs continuous batching on the same GPU

import random
# One "step" = generate one token for every request in the batch right now.
random.seed(7)
NUM_REQUESTS = 60
BATCH_SIZE   = 8                 # how many requests fit in the batch at once
# Output lengths vary a lot: some 1-liners, some long answers.
lengths = [random.choice([12, 20, 40, 80, 160]) for _ in range(NUM_REQUESTS)]

def static_batching(lengths, batch_size):
    """Fill a batch, run it to completion, then start the next batch.
    The whole batch is blocked until its SLOWEST request finishes."""
    steps = 0; completion = []
    for start in range(0, len(lengths), batch_size):
        group = lengths[start:start + batch_size]
        group_steps = max(group)     # everyone waits for the longest one
        for _ in group:
            completion.append(steps + group_steps)
        steps += group_steps
    return steps, completion

def continuous_batching(lengths, batch_size):
    """The moment one request finishes, a waiting one takes its slot on the
    next step. The batch stays full, the GPU stays busy."""
    waiting = list(lengths); active = []; completion = []; steps = 0
    while waiting or active:
        while len(active) < batch_size and waiting:
            active.append(waiting.pop(0))    # admit into free slots
        steps += 1
        nxt = []
        for remaining in active:
            remaining -= 1                   # one token generated this step
            if remaining == 0:
                completion.append(steps)
            else:
                nxt.append(remaining)
        active = nxt
    return steps, completion

for name, fn in [("static batching", static_batching),
                 ("continuous batching", continuous_batching)]:
    total_steps, completion = fn(lengths, BATCH_SIZE)
    throughput = NUM_REQUESTS / total_steps
    avg_latency = sum(completion) / len(completion)
    print(f"{name:<22} steps={total_steps:>4}  "
          f"throughput={throughput:5.3f} req/step  "
          f"avg latency={avg_latency:6.1f} steps")

s_steps, _ = static_batching(lengths, BATCH_SIZE)
c_steps, _ = continuous_batching(lengths, BATCH_SIZE)
print(f"\nContinuous batching finished the same {NUM_REQUESTS} requests "
      f"{(s_steps / c_steps - 1) * 100:.0f}% faster on the same GPU.")

▶ Output

static batching        steps=1280  throughput=0.047 req/step  avg latency= 682.7 steps
continuous batching    steps= 592  throughput=0.101 req/step  avg latency= 248.7 steps

Continuous batching finished the same 60 requests 116% faster on the same GPU.

What happened here: same GPU, same 60 requests, same batch size, and continuous batching finished more than twice as fast with less than half the average latency. The reason is the mixed output lengths. Under static batching, a batch stuffed with one 160-token answer forces seven short requests to sit and wait for it. Continuous batching lets those short ones finish and free their slots for waiting work immediately, so the batch is never half-empty.

This is a simplified model (it ignores memory limits and the prefill step) but it captures the exact win vLLM ships. To actually run it, you install vLLM on a GPU box and start vllm serve, which again exposes an OpenAI-compatible endpoint, so the same one-line base_url swap from the Ollama section points your app at it.

🖥️ terminal: serving with vLLM on a GPU box (OpenAI-compatible)

pip install vllm
# Starts an OpenAI-compatible server on port 8000 with PagedAttention +
# continuous batching handled for you. Needs an NVIDIA GPU.
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192

# Your app then points at it, same client as before:
#   client = OpenAI(base_url="http://localhost:8000/v1", api_key="vllm")

KV-Cache Math: The Real Memory Budget

Here is the interview question that separates people who have served a model from people who have only called an API: how big is the KV cache, and why does it, not the weights, cap your batch size? The answer is pure arithmetic, so let us work it out. For every token the model has seen, it stores one key vector and one value vector, in every layer. The size per token is therefore two (key plus value) times the number of layers times the KV width per layer times the bytes per number. Say Aviraj is sizing a serving box for a typical 8B model that uses grouped-query attention, which shrinks the KV width by sharing key/value heads.

📄 kv_cache_math.py: bytes per token, then how many users fit

# Bytes per token = 2 (one Key + one Value)
#                 x num_layers
#                 x num_kv_heads x head_dim   (the KV width per layer)
#                 x bytes_per_element (2 for fp16)
num_layers   = 32
num_kv_heads = 8      # grouped-query attention: fewer KV heads than 32 attn heads
head_dim     = 128
bytes_per_el = 2      # fp16 cache

per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_el
print(f"KV cache per token: {per_token:,} bytes = {per_token/1024:.0f} KiB")

context_len = 8192                       # one request that fills an 8K context
per_request = per_token * context_len
print(f"One {context_len}-token request: {per_request/1024**3:.2f} GiB of KV cache")

# The squeeze: a 24 GB GPU after the fp16 weights are loaded.
gpu_total_gb = 24
weights_gb   = 16                        # 8B in fp16 (~15 GiB), rounded up
kv_budget_gb = gpu_total_gb - weights_gb
max_full_ctx = (kv_budget_gb * 1024**3) // per_request
print(f"\nGPU: {gpu_total_gb} GB total, ~{weights_gb} GB weights, "
      f"{kv_budget_gb} GB left for KV")
print(f"Concurrent 8K-context requests that fit: {int(max_full_ctx)}")
print("This is why the KV cache, not the weights, caps your batch size.")

▶ Output

KV cache per token: 131,072 bytes = 128 KiB
One 8192-token request: 1.00 GiB of KV cache

GPU: 24 GB total, ~16 GB weights, 8 GB left for KV
Concurrent 8K-context requests that fit: 8
This is why the KV cache, not the weights, caps your batch size.

What happened here: each token costs 128 KiB of cache, so one request at full 8K context eats a whole gigabyte, and after the weights there is only room for eight such requests on a 24 GB card. That is the ceiling nobody warns you about. It also explains the two big levers you have. Quantizing the KV cache to 8-bit halves that per-token cost and doubles how many users fit, which is why serving engines expose a KV-cache dtype setting.

And PagedAttention wins precisely because most requests do not use the full 8K, so paging reclaims the unused portion and packs in far more than eight. On the speed side, the two levers to name in an interview are batching (already covered, it raises throughput per GPU) and speculative decoding, where a small draft model proposes several tokens that the big model verifies in one pass, cutting latency. The KV cache is the resource; batching and speculation are how you spend it well.

Choosing an LLM Serving Stack

You do not need a different tool for every job, but you do need the right one for each stage. The honest short version: develop on Ollama, ship to production on vLLM, run on CPU or edge with llama.cpp, and orchestrate on Kubernetes with KServe wrapping a vLLM runtime. Here is the decision table, with the reason each choice wins, so you can point at it in a design review.

A quick note on tools you will see in older LLM serving tutorials, so a stale blog post does not send you down the wrong path. TorchServe, PyTorch’s own model server, was archived in 2025 and should be treated as history, not a starting point. Text Generation Inference (TGI) from Hugging Face was a genuine early leader but is in light maintenance now, with the community having largely moved to vLLM and SGLang for new work. The concepts (a serving loop, batching, an inference runtime) outlast all of these names, which is why we anchored the post on the KV cache and batching rather than on any one binary.

Common Mistakes

Best Practices

Frequently Asked Questions

Interview Questions on LLM Serving

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: Why does the KV cache, not the weights, cap how many users a GPU can hold?

The weights are a fixed cost paid once, but every active request grows its own KV cache with each generated token. At long context lengths, a few dozen concurrent conversations can eat more memory than the model itself, so concurrency is capped by cache, not weights. That is why production engines page the cache into blocks, and why capping the maximum context length frees room for more users.

Q: Explain continuous batching and why it beats static batching.

Static batching waits to fill a batch and then holds it until the longest answer finishes, so short requests idle behind long ones. Continuous batching admits and retires requests token by token: when one sequence finishes, another takes its slot on the very next step. The GPU stays busy under real mixed-length traffic, which is where most of vLLM’s throughput advantage comes from.

Q: When would you serve a 4-bit model instead of fp16?

When memory or hardware is the binding constraint and a small quality loss is acceptable: laptops, edge boxes, or fitting a bigger model onto a smaller GPU. 8-bit is nearly free on quality and halves memory; 4-bit accepts a modest quality dip to quarter it. On a large GPU where quality is the priority, fp16 or 8-bit stays the default.

Q: A team ships Ollama to production and latency collapses under load. What happened?

Ollama is a development server: excellent for one user, not designed for high-concurrency batching. Under parallel load, requests queue up and tail latency explodes even though each request alone looks fine. The fix is moving to an engine built for concurrency such as vLLM, keeping the same OpenAI-compatible client, and doing the KV-cache math to size the GPU for the target number of users.

Want more? Hugging Face documentation documents everything this post could not fit.

What Comes Next

You now have the whole LLM serving path: Ollama for private local development, quantization matched to the hardware you own, vLLM with PagedAttention and continuous batching for production traffic, and the KV-cache math that turns GPU shopping from guesswork into arithmetic. The next stop widens the lens from one model server to deploying full ML systems at scale. For every topic in order, visit the Python + AI/ML tutorial series home.

Related Posts

Previous: LLM and GenAI Interview Questions: 40-Question Checkpoint

Next: Python ML Production Deployment at Scale: FastAPI, Docker, Cloud

Series Home: Python + AI/ML Tutorial Series