A few years ago, the idea that large language models could write a passable email felt like science fiction. Today they write production code, reason through tricky math, and hold conversations that fool experienced engineers. The strangest part: nobody programmed those abilities in. They appeared once models crossed a size threshold, and this guide maps how that happened and what it means for your projects.

You already understand Transformers and attention, plus BERT vs GPT, from the earlier lessons. Now we zoom out. What makes an LLM different from the language models you have already built? Why does scale matter so much? And how do you actually think about choosing and using these models in your own projects?

Here is a quick way to picture scale. A small language model is like a new hire who memorized a phrasebook: ask a simple question and you get a tidy canned answer, but anything off-script and they freeze. A large model is like a senior engineer who has read the whole company wiki, every old ticket, and most of the internet: you can hand them a vague problem and they reason their way to something useful. Same job title, wildly different capability, and the only thing that changed was how much they read. That is what scaling does to a language model.

“The most surprising thing about large language models is not what they were trained to do, but what they learned to do along the way.”

Jason Wei, Google Brain

Last Updated: July 2026 | Tested on: Python 3.14.6, tiktoken 0.13 | Difficulty: Intermediate | Reading Time: 30 minutes

A large language model (LLM) is a neural network trained on massive text datasets that can generate, understand, and reason about natural language. What separates LLMs from earlier language models is scale (billions of parameters trained on trillions of tokens) and the emergent abilities that appear at scale: few-shot learning, chain-of-thought reasoning, and code generation that nobody explicitly programmed.

This post maps the LLM landscape: what makes them work, why scaling laws matter, the key model families (GPT, Claude, Llama, Gemini), and how to think about choosing and using them in your own projects. Model versions change every few months, so we focus on the families and the patterns, not on memorizing a single release. Understanding LLMs at this level is essential before you start building with their APIs.

By the end, you’ll understand tokenization, context windows, temperature, the difference between base and instruction-tuned models, and the practical trade-offs that determine which model fits your use case.

How LLMs Scale From Millions to Trillions

The diagram below shows the key insight: LLMs are not a new architecture. They are the same Transformer you learned in transformer architecture tutorial, but scaled up by orders of magnitude in parameters, training data, and compute. That scale unlocks qualitatively different behavior.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows how large language models are built in stages. Pre-training on massive text corpora teaches the model language structure. Supervised fine-tuning on curated examples teaches it to follow instructions. RLHF (reinforcement learning from human feedback) aligns its outputs with human preferences. Each stage builds on the previous one, which is why foundation models are so expensive to create from scratch but relatively cheap to adapt. Understanding these stages helps you pick the right approach for your use case. Most applications need only the final Application Programming Interface (API), not the full training pipeline.

What Makes a Language Model “Large”?

Every language model you have seen so far (BERT-base with 110 million parameters, GPT-2 with 1.5 billion) counts as a language model. But the term “Large Language Model” usually means a model with tens of billions of parameters or more. The line is not just marketing. Something genuinely changes once you scale up.

Small models learn pattern matching: given these tokens, predict the next one. Large models still do that, but they also develop what researchers call emergent abilities, capabilities that appear suddenly once a model crosses a parameter threshold. A 1-billion parameter model cannot do chain-of-thought reasoning. A 100-billion parameter model can, even though nobody trained it to reason step by step. It learned that ability as a side effect of learning to predict text at massive scale.

Think of a settlement. A village of a few hundred people cannot sustain a hospital, a university, or a symphony orchestra. Grow it into a city of a few million and those institutions appear on their own, not because anyone forced them into being, but because the sheer scale finally makes them possible. Language models behave the same way: certain abilities only become possible once the model is big enough. The three ingredients that define an LLM are parameter count (model size), training data (what the model has read), and compute (how long it trained). Scaling any one of these helps. Scaling all three together produces the dramatic improvements we have seen since 2022.

The Modern LLM Landscape

Think of it like choosing where dinner comes from. A fine-dining restaurant (a closed API) hands you a polished plate but charges you for every course, while cooking at home from your own pantry (open weights) takes more effort up front but lets you control every ingredient and pay nothing per serving. The LLM space moves fast, but the major players have settled into clear categories like these. Getting the categories straight matters far more than memorizing one model version, because the versions change every few months. The exact model names below are current at the time of writing; treat them as examples of each tier, and always check the provider docs for today’s lineup.

📄 llm_landscape.py: mapping the LLM ecosystem

# Rahul maps out the LLM landscape for his team
llm_landscape = {
    "Closed-Source (API-only)": {
        "OpenAI GPT-5.5": {
            "params": "Undisclosed (MoE architecture)",
            "strengths": "Multimodal, fast, broad knowledge",
            "access": "API only (see openai.com/pricing for current rates)",
        },
        "Anthropic Claude Opus 4.8": {
            "params": "Undisclosed",
            "strengths": "Long context (1M tokens), strong reasoning, safety-focused",
            "access": "API only (see anthropic.com/pricing for current rates)",
        },
        "Google Gemini 3.1 Pro": {
            "params": "Undisclosed (MoE architecture)",
            "strengths": "Multimodal native, large context, code generation",
            "access": "API + Google AI Studio",
        },
    },
    "Open-Source / Open-Weight": {
        "Meta Llama 3.1 (405B)": {
            "params": "405 billion",
            "strengths": "Strong open model, multilingual, tool use",
            "access": "Download weights, run locally or cloud",
        },
        "Mistral Large 2": {
            "params": "123 billion",
            "strengths": "Efficient, strong for size, code generation",
            "access": "Open weights + API",
        },
        "Qwen 2.5 (72B)": {
            "params": "72 billion",
            "strengths": "Multilingual, math, coding",
            "access": "Open weights (Apache 2.0)",
        },
    },
    "Small but Capable (runs on laptop)": {
        "Llama 3.2 (3B)": {
            "params": "3 billion",
            "strengths": "Mobile-friendly, fast inference",
            "access": "Download, run with llama.cpp or Ollama",
        },
        "Phi-3.5 Mini": {
            "params": "3.8 billion",
            "strengths": "Reasoning beyond its size class",
            "access": "Open weights (MIT license)",
        },
    },
}

# Display the landscape
for category, models in llm_landscape.items():
    print(f"\n{'='*60}")
    print(f"  {category}")
    print(f"{'='*60}")
    for name, info in models.items():
        print(f"\n  {name}")
        for key, val in info.items():
            print(f"    {key:>12}: {val}")

▶ Output

============================================================
  Closed-Source (API-only)
============================================================

  OpenAI GPT-5.5
          params: Undisclosed (MoE architecture)
       strengths: Multimodal, fast, broad knowledge
          access: API only (see openai.com/pricing for current rates)

  Anthropic Claude Opus 4.8
          params: Undisclosed
       strengths: Long context (1M tokens), strong reasoning, safety-focused
          access: API only (see anthropic.com/pricing for current rates)

  Google Gemini 3.1 Pro
          params: Undisclosed (MoE architecture)
       strengths: Multimodal native, large context, code generation
          access: API + Google AI Studio

============================================================
  Open-Source / Open-Weight
============================================================

  Meta Llama 3.1 (405B)
          params: 405 billion
       strengths: Strong open model, multilingual, tool use
          access: Download weights, run locally or cloud

  Mistral Large 2
          params: 123 billion
       strengths: Efficient, strong for size, code generation
          access: Open weights + API

  Qwen 2.5 (72B)
          params: 72 billion
       strengths: Multilingual, math, coding
          access: Open weights (Apache 2.0)

============================================================
  Small but Capable (runs on laptop)
============================================================

  Llama 3.2 (3B)
          params: 3 billion
       strengths: Mobile-friendly, fast inference
          access: Download, run with llama.cpp or Ollama

  Phi-3.5 Mini
          params: 3.8 billion
       strengths: Reasoning beyond its size class
          access: Open weights (MIT license)

What happened here: We sorted the LLM landscape into three tiers by how you access them and what they can do. Closed-source models (GPT, Claude, Gemini) lead on raw capability but require API calls and cost money per token. Open-weight models (Llama, Mistral, Qwen) let you run inference on your own hardware, which matters for privacy, cost control, and customization. Small models (3 to 4 billion parameters) run on a laptop and are surprisingly capable for focused tasks. The names will change over time, but the three tiers stay put, so learn to think in tiers first.

Emergent Abilities: When Scale Creates Surprise

The most fascinating thing about LLMs is emergence. Below a certain scale, a model simply cannot do a task. Above that scale, it can, with no change to the architecture at all. Nobody programmed GPT-3 to do arithmetic, translate between languages it was barely trained on, or write Python code. Those abilities just showed up, squeezed out of the sheer volume of text patterns the model swallowed. Think of it like learning to cook: read enough recipes and one day you can improvise a new dish nobody handed you a recipe for.

📄 emergent_abilities.py: what scale unlocks

# Viraj demonstrates emergent abilities at different scales
emergent_capabilities = {
    "~1B parameters": {
        "can_do": ["Text completion", "Basic grammar", "Simple Q&A"],
        "cannot_do": ["Chain-of-thought reasoning", "Code generation", "Math"],
        "example_models": ["GPT-2", "Phi-1"],
    },
    "~10B parameters": {
        "can_do": ["Summarization", "Translation", "Simple code", "Few-shot learning"],
        "cannot_do": ["Complex reasoning", "Multi-step math", "Nuanced analysis"],
        "example_models": ["Llama 3.2 (8B)", "Mistral 7B"],
    },
    "~70B+ parameters": {
        "can_do": [
            "Chain-of-thought reasoning",
            "Complex code generation",
            "Multi-step math",
            "Creative writing",
            "Tool use / function calling",
            "Self-correction",
        ],
        "cannot_do": ["Perfect factual accuracy (still hallucinates)"],
        "example_models": ["Llama 3.1 (70B)", "Qwen 2.5 (72B)"],
    },
    "~200B+ parameters": {
        "can_do": [
            "Everything above, plus:",
            "Expert-level reasoning",
            "PhD-level science questions",
            "Multi-turn planning",
            "Agentic behavior",
        ],
        "cannot_do": ["Guaranteed truthfulness", "Real-time knowledge"],
        "example_models": ["GPT-5.5", "Claude Opus 4.8", "Gemini 3.1 Pro"],
    },
}

for scale, info in emergent_capabilities.items():
    print(f"\n{'─'*50}")
    print(f"  Scale: {scale}")
    print(f"  Models: {', '.join(info['example_models'])}")
    print(f"  CAN do:")
    for ability in info["can_do"]:
        print(f"    ✓ {ability}")
    print(f"  CANNOT do:")
    for limitation in info["cannot_do"]:
        print(f"    ✗ {limitation}")

▶ Output

──────────────────────────────────────────────────
  Scale: ~1B parameters
  Models: GPT-2, Phi-1
  CAN do:
    ✓ Text completion
    ✓ Basic grammar
    ✓ Simple Q&A
  CANNOT do:
    ✗ Chain-of-thought reasoning
    ✗ Code generation
    ✗ Math

──────────────────────────────────────────────────
  Scale: ~10B parameters
  Models: Llama 3.2 (8B), Mistral 7B
  CAN do:
    ✓ Summarization
    ✓ Translation
    ✓ Simple code
    ✓ Few-shot learning
  CANNOT do:
    ✗ Complex reasoning
    ✗ Multi-step math
    ✗ Nuanced analysis

──────────────────────────────────────────────────
  Scale: ~70B+ parameters
  Models: Llama 3.1 (70B), Qwen 2.5 (72B)
  CAN do:
    ✓ Chain-of-thought reasoning
    ✓ Complex code generation
    ✓ Multi-step math
    ✓ Creative writing
    ✓ Tool use / function calling
    ✓ Self-correction
  CANNOT do:
    ✗ Perfect factual accuracy (still hallucinates)

──────────────────────────────────────────────────
  Scale: ~200B+ parameters
  Models: GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro
  CAN do:
    ✓ Everything above, plus:
    ✓ Expert-level reasoning
    ✓ PhD-level science questions
    ✓ Multi-turn planning
    ✓ Agentic behavior
  CANNOT do:
    ✗ Guaranteed truthfulness
    ✗ Real-time knowledge

What happened here: We mapped emergent abilities to parameter scale. The key idea is that these abilities are not a smooth ramp. They tend to appear suddenly once a model passes a size threshold. A 7B model that cannot do chain-of-thought reasoning is not just “a little worse” at it. It genuinely cannot do it. Scale past the threshold and the ability shows up. One honest caveat: researchers still debate how sharp these jumps really are, and some of the sharpness comes from the all-or-nothing way we score the tasks. The big picture holds though: scale buys you abilities you never explicitly asked for.

Tokenization and Context Windows

You learned about tokenization in the Hugging Face tutorial with BERT’s WordPiece. LLMs use similar subword tokenization (usually BPE, short for Byte Pair Encoding), but the context window is far bigger. BERT handles 512 tokens. Modern LLMs handle 128K to a million-plus tokens. That gap completely changes what you can do. Instead of feeding a model one paragraph, you can hand it an entire codebase, a full textbook chapter, or thousands of customer reviews at once. The context window is the model’s short-term memory: think of it as how much it can hold in its head before something has to fall out the other side.

📄 context_windows.py: token limits across models

# Aditi compares context windows across models
import tiktoken  # OpenAI's tokenizer library

context_windows = {
    "BERT (2018)":          512,
    "GPT-2 (2019)":         1_024,
    "GPT-3 (2020)":         4_096,
    "GPT-3.5 (2022)":       4_096,
    "GPT-4 (2023)":         8_192,
    "GPT-4 Turbo (2023)":   128_000,
    "Claude 3.5 (2024)":    200_000,
    "Claude Opus (2025)": 1_000_000,
    "Gemini 2.5 (2025)":    2_000_000,
}

# What fits in each context window?
# Rule of thumb: 1 token ≈ 0.75 English words
print(f"{'Model':<25} {'Tokens':>12} {'~Words':>12} {'~Pages':>8}")
print(f"{'─'*25} {'─'*12} {'─'*12} {'─'*8}")

for model, tokens in context_windows.items():
    words = int(tokens * 0.75)
    pages = words // 250  # ~250 words per page
    print(f"{model:<25} {tokens:>12,} {words:>12,} {pages:>8,}")

# Demonstrate tokenization with tiktoken
print(f"\n--- BPE Tokenization (GPT-4 tokenizer) ---")
enc = tiktoken.encoding_for_model("gpt-4")

texts = [
    "Hello, world!",
    "Viraj is building a RAG application with Python.",
    "def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)",
]

for text in texts:
    tokens = enc.encode(text)
    print(f"\n  Text:   {text}")
    print(f"  Tokens: {len(tokens)} -> {tokens[:10]}{'...' if len(tokens) > 10 else ''}")

▶ Output

Model                           Tokens       ~Words   ~Pages
───────────────────────── ──────────── ──────────── ────────
BERT (2018)                        512          384        1
GPT-2 (2019)                     1,024          768        3
GPT-3 (2020)                     4,096        3,072       12
GPT-3.5 (2022)                   4,096        3,072       12
GPT-4 (2023)                     8,192        6,144       24
GPT-4 Turbo (2023)             128,000       96,000      384
Claude 3.5 (2024)              200,000      150,000      600
Claude Opus (2025)           1,000,000      750,000    3,000
Gemini 2.5 (2025)            2,000,000    1,500,000    6,000

--- BPE Tokenization (GPT-4 tokenizer) ---

  Text:   Hello, world!
  Tokens: 4 -> [9906, 11, 1917, 0]

  Text:   Viraj is building a RAG application with Python.
  Tokens: 11 -> [50135, 1662, 374, 4857, 264, 432, 1929, 3851, 449, 13325]...

  Text:   def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)
  Tokens: 23 -> [755, 76798, 1471, 1680, 471, 308, 422, 308, 2717, 220]...

What happened here: Context windows have grown by thousands of times since BERT. A million-token window can hold roughly 3,000 pages of text in a single prompt. That is an entire novel series. BPE tokenization breaks text into subword units: “Hello, world!” becomes 4 tokens, while that one-line Python function becomes 23. Code usually costs more tokens than plain English prose because variable names, punctuation, and syntax create more subword splits. The token IDs you see are just lookups into the tokenizer’s vocabulary, so they will match exactly when you run this with the same gpt-4 encoding.

What Actually Happens When You Send a Prompt

You have now seen the pieces one at a time: tokens, the context window, temperature. Here is the whole trip your prompt takes from the moment you hit send, laid out as a single flow. Imagine you type “Write a haiku about coffee” and press enter. This is what the model does with it, step by step.

1. Your text is split into tokens (the subword chunks from the previous section), and each token is turned into a number the model can work with.
2. Those numbers pass through the model one time. That single sweep from input to output is called a forward pass.
3. At the end of that pass, the model produces a probability for every possible next token in its vocabulary, a long list of how likely each one is to come next.
4. One token is picked by sampling from that probability list. The temperature setting controls how adventurous the pick is: near 0 means it almost always takes the top choice, while a higher temperature gives lower-ranked tokens a real chance and adds variety.
5. The chosen token is added to the text, and the whole thing runs again to produce the next token, then the next, one at a time. This loop, where each output is fed back in as new input, is called autoregressive generation.
6. Each new token is streamed back to you the moment it is produced, which is why the reply appears on your screen word by word instead of arriving all at once.
7. Finally the token numbers are turned back into readable characters, a step called detokenization, and you see plain text.

Notice what step 4 quietly implies. Because the next token is sampled from a list of probabilities rather than fixed in advance, the same prompt can give you different answers on different runs. Set the temperature near 0 and the output becomes almost deterministic, close to the same reply every time; raise the temperature and you invite more variety. Autoregressive generation itself is just writing a sentence one word at a time, where each new word is chosen by looking at everything written so far, so the model is really finishing its own sentence over and over until the reply is done.

Beyond Temperature: Top-k, Top-p, and Greedy Decoding

Step 4 above said one token gets picked from the probability list, but it skipped over the interesting part: how. That choosing step has a name, decoding, and there are a few common strategies for it. They all start from the same list of probabilities the model just produced, and they differ only in which tokens they let into the running and how they gamble among them. Getting a feel for the three you will meet most often (greedy, top-k, and top-p) explains most of the sampling knobs you find in an LLM API.

The simplest strategy is greedy decoding: at every step, just take the single highest-probability token and move on. It is fast and completely deterministic, so the same prompt always gives the same answer. The downside is that always grabbing the safest next word tends to paint the model into a corner. It can start repeating itself or settle into flat, predictable phrasing, a bit like a speaker who only ever reaches for the most obvious word and never once surprises you.

Top-k sampling loosens that up. Instead of always taking the top token, you keep the k most likely tokens (say the top 40), throw the rest away, and then pick randomly among the survivors in proportion to their probabilities. Now the model has room to choose a slightly less obvious word now and then, which reads as more natural and less robotic. The catch is that k is a fixed count. Keeping 40 candidates is generous when the model is unsure and dozens of words are plausible, but wasteful when the model is confident and only two or three words really make sense.

Top-p sampling, also called nucleus sampling, fixes that rigidity. Rather than keeping a fixed number, you keep the smallest set of top tokens whose probabilities add up to at least p, for example 0.9, and then sample among just those. The clever part is that the pool resizes itself with the model’s confidence. When the model is sure, a couple of tokens already cover 90 percent of the probability and the pool stays tiny, close to greedy. When the model is genuinely uncertain, it takes many tokens to reach that 0.9, so the pool widens and lets in more variety. You are setting a confidence budget rather than a headcount, and the model spends it as the situation demands.

Where does temperature fit in all this? It runs first, before any of the above trims the list. Temperature reshapes the probability distribution itself: a low temperature (near 0) exaggerates the gaps so the top token towers over the rest and the pick becomes nearly greedy, while a high temperature flattens the distribution so unlikely tokens get a more even shot. So temperature decides how peaked or flat the odds are, and top-k or top-p then decide which slice of that reshaped list you are allowed to sample from.

A simple rule of thumb keeps you out of trouble: tune temperature or top_p, but do not crank both hard at the same time, since they push on the same randomness and stack in ways that are tough to reason about. Reach for lower values (or plain greedy) when you want factual, repeatable answers like data extraction or code, and raise them when you want creative range like brainstorming or story writing. Start from the provider’s defaults, change one knob at a time, and watch what it does to your output.

How LLMs Are Trained

Training an LLM happens in three distinct phases, each one more focused than the last. Understanding this pipeline explains a lot about why LLMs behave the way they do, including their habit of hallucinating confident-sounding nonsense. A useful way to picture it: pre-training is like a student reading the entire library, fine-tuning is like an internship where they learn to actually answer questions, and alignment is like the feedback from a good mentor on how to be helpful and not say anything reckless.

📄 training_pipeline.py: the three phases of LLM training

# Aviraj documents the LLM training pipeline
training_phases = [
    {
        "phase": "1. Pre-training",
        "objective": "Next token prediction on massive text corpus",
        "data": "Trillions of tokens from internet, books, code, Wikipedia",
        "compute": "Thousands of GPUs for weeks to months",
        "cost": "$10M - $100M+ for frontier models",
        "what_model_learns": [
            "Grammar and syntax of 100+ languages",
            "World knowledge (facts, history, science)",
            "Reasoning patterns (from math/code)",
            "Code syntax for dozens of programming languages",
        ],
        "limitation": "Model is a text completion engine, not helpful, not safe",
    },
    {
        "phase": "2. Supervised Fine-Tuning (SFT)",
        "objective": "Learn to follow instructions and be helpful",
        "data": "100K-1M human-written instruction/response pairs",
        "compute": "Days on ~100 GPUs",
        "cost": "$100K - $1M",
        "what_model_learns": [
            "How to interpret and follow instructions",
            "Conversational format (multi-turn)",
            "Task-specific patterns (summarize, translate, code)",
        ],
        "limitation": "Helpful but may produce harmful, biased, or wrong content",
    },
    {
        "phase": "3. RLHF / Constitutional AI / DPO",
        "objective": "Align model with human preferences and safety",
        "data": "Human preference rankings (which response is better?)",
        "compute": "Days on ~100 GPUs",
        "cost": "$100K - $500K",
        "what_model_learns": [
            "Which responses humans prefer",
            "When to refuse harmful requests",
            "How to express uncertainty",
            "Tone, helpfulness, factual grounding",
        ],
        "limitation": "Can be over-cautious; alignment is never perfect",
    },
]

for phase in training_phases:
    print(f"\n{'='*55}")
    print(f"  {phase['phase']}")
    print(f"{'='*55}")
    print(f"  Objective: {phase['objective']}")
    print(f"  Data:      {phase['data']}")
    print(f"  Compute:   {phase['compute']}")
    print(f"  Cost:      {phase['cost']}")
    print(f"  Learns:")
    for item in phase["what_model_learns"]:
        print(f"    -> {item}")
    print(f"  Limitation: {phase['limitation']}")

▶ Output

=======================================================
  1. Pre-training
=======================================================
  Objective: Next token prediction on massive text corpus
  Data:      Trillions of tokens from internet, books, code, Wikipedia
  Compute:   Thousands of GPUs for weeks to months
  Cost:      $10M - $100M+ for frontier models
  Learns:
    -> Grammar and syntax of 100+ languages
    -> World knowledge (facts, history, science)
    -> Reasoning patterns (from math/code)
    -> Code syntax for dozens of programming languages
  Limitation: Model is a text completion engine, not helpful, not safe

=======================================================
  2. Supervised Fine-Tuning (SFT)
=======================================================
  Objective: Learn to follow instructions and be helpful
  Data:      100K-1M human-written instruction/response pairs
  Compute:   Days on ~100 GPUs
  Cost:      $100K - $1M
  Learns:
    -> How to interpret and follow instructions
    -> Conversational format (multi-turn)
    -> Task-specific patterns (summarize, translate, code)
  Limitation: Helpful but may produce harmful, biased, or wrong content

=======================================================
  3. RLHF / Constitutional AI / DPO
=======================================================
  Objective: Align model with human preferences and safety
  Data:      Human preference rankings (which response is better?)
  Compute:   Days on ~100 GPUs
  Cost:      $100K - $500K
  Learns:
    -> Which responses humans prefer
    -> When to refuse harmful requests
    -> How to express uncertainty
    -> Tone, helpfulness, factual grounding
  Limitation: Can be over-cautious; alignment is never perfect

What happened here: LLM training is a three-stage pipeline. Pre-training creates raw capability (expensive: tens of millions of dollars). Supervised fine-tuning (SFT) teaches the model to be helpful (moderate cost). Alignment (RLHF, Constitutional AI, or DPO) teaches it to be safe and match human preferences (moderate cost). This explains why base models like Llama exist alongside instruction-tuned variants like Llama-Chat: the base model has the knowledge but is not helpful, while the chat variant has been through all three stages.

Running LLMs Efficiently

A 70-billion parameter model in full 32-bit precision needs around 260 GB of memory just to load the weights. That is several high-end GPUs. Quantization shrinks models by storing each parameter in fewer bits, trading a small accuracy loss for a big drop in memory and faster inference. This is the trick that lets people run real LLMs on a laptop. Think of it like saving a photo as a JPEG instead of a giant RAW file: you lose a tiny bit of detail most people will never notice, and the file gets small enough to actually carry around.

📄 quantization_math.py: why quantization makes LLMs practical

# Prathamesh calculates memory requirements at different precisions
model_params = {
    "Llama 3.2 (3B)": 3e9,
    "Mistral 7B": 7e9,
    "Llama 3.1 (70B)": 70e9,
    "Llama 3.1 (405B)": 405e9,
}

precisions = {
    "FP32 (32-bit)": 4,    # 4 bytes per param
    "FP16 (16-bit)": 2,    # 2 bytes per param
    "INT8 (8-bit)":  1,    # 1 byte per param
    "INT4 (4-bit)":  0.5,  # 0.5 bytes per param
}

print(f"{'Model':<22}", end="")
for prec in precisions:
    print(f" {prec:>16}", end="")
print()
print("─" * 88)

for model_name, params in model_params.items():
    print(f"{model_name:<22}", end="")
    for prec_name, bytes_per in precisions.items():
        gb = (params * bytes_per) / (1024**3)
        print(f" {gb:>13.1f} GB", end="")
    print()

print(f"\n--- What runs where? ---")
hardware = [
    ("Laptop (16 GB RAM)", "16 GB", "3B @ INT4, 7B @ INT4"),
    ("Gaming PC (24 GB VRAM)", "24 GB", "7B @ FP16, 13B @ INT4"),
    ("Single A100 (80 GB)", "80 GB", "70B @ INT4, 13B @ FP16"),
    ("8x A100 cluster (640 GB)", "640 GB", "70B @ FP16, 405B @ INT4"),
]

print(f"\n{'Hardware':<30} {'VRAM':>8} {'Can Run':>40}")
print("─" * 80)
for hw, vram, models in hardware:
    print(f"{hw:<30} {vram:>8} {models:>40}")

▶ Output

Model                     FP32 (32-bit)    FP16 (16-bit)     INT8 (8-bit)     INT4 (4-bit)
────────────────────────────────────────────────────────────────────────────────────────
Llama 3.2 (3B)                  11.2 GB           5.6 GB           2.8 GB           1.4 GB
Mistral 7B                      26.1 GB          13.0 GB           6.5 GB           3.3 GB
Llama 3.1 (70B)                260.8 GB         130.4 GB          65.2 GB          32.6 GB
Llama 3.1 (405B)              1508.7 GB         754.4 GB         377.2 GB         188.6 GB

--- What runs where? ---

Hardware                           VRAM                                  Can Run
────────────────────────────────────────────────────────────────────────────────
Laptop (16 GB RAM)                16 GB                     3B @ INT4, 7B @ INT4
Gaming PC (24 GB VRAM)            24 GB                    7B @ FP16, 13B @ INT4
Single A100 (80 GB)               80 GB                   70B @ INT4, 13B @ FP16
8x A100 cluster (640 GB)         640 GB                  70B @ FP16, 405B @ INT4

What happened here: Quantization from FP32 to INT4 reduces memory by 8x. A 70B model that needs 260 GB at full precision fits in 33 GB at 4-bit quantization, within reach of a single high-end Graphics Processing Unit (GPU). Tools like llama.cpp, GPTQ, and bitsandbytes make quantization practical. The accuracy loss from INT4 quantization is typically 1-3% on benchmarks, which is a remarkable trade-off for making models runnable on consumer hardware.

Running an LLM Locally with Ollama

You do not need an API key or a cloud GPU to start playing with LLMs. Ollama is an open-source tool that downloads, quantizes, and serves open models right on your own machine. It takes about five minutes to set up and gives you a local LLM API that looks a lot like the OpenAI format. The snippet below talks to a model running on your laptop, so the output you get depends on which model you pulled.

📄 ollama_local.py: running LLMs on your own machine

# Step 1: Install Ollama from https://ollama.com
# Step 2: Pull a model (in terminal):
#   ollama pull llama3.2:3b
# Step 3: Use the Python API

import requests
import json

# Anvi queries a local LLM via Ollama's REST API
def chat_local(prompt, model="llama3.2:3b"):
    """Send a prompt to locally-running Ollama."""
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": model,
            "prompt": prompt,
            "stream": False,
            "options": {
                "temperature": 0.7,
                "num_predict": 200,
            },
        },
    )
    return response.json()["response"]

# Ask the local model to explain a concept
answer = chat_local(
    "Explain what a Python decorator is in 3 sentences. "
    "Include a simple code example."
)
print("Local LLM response:")
print(answer)

# Check model info
info = requests.get("http://localhost:11434/api/tags").json()
for model in info["models"]:
    size_gb = model["size"] / (1024**3)
    print(f"\nModel: {model['name']}")
    print(f"  Size: {size_gb:.1f} GB")
    print(f"  Modified: {model['modified_at'][:10]}")

▶ Output (example, your local model will word it differently)

Local LLM response:
A Python decorator is a function that wraps another function to extend its
behavior without modifying its source code. It uses the @decorator syntax
placed above the function definition. Here's a simple example:

```python
def timer(func):
    import time
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.time() - start:.2f}s")
        return result
    return wrapper

@timer
def slow_function():
    time.sleep(1)
```

Model: llama3.2:3b
  Size: 1.9 GB
  Modified: 2025-12-15

What happened here: This snippet runs a 3-billion parameter LLM entirely on a local machine: no internet connection, no API key, and no cost per token. Ollama serves models through a REST API and also exposes an OpenAI-compatible endpoint, so the same client code you write for the cloud often works against your laptop (we lean on that in the LLM API and function calling tutorial). The 3B model at 4-bit quantization takes under 2 GB of disk space and answers in seconds on a modern laptop.

Because the model itself produces the text, the exact wording changes every run, which is why the output above is labeled an example. For learning, prototyping, and privacy-sensitive work, local LLMs are hard to beat.

Choosing the Right LLM for Your Task

Picking a model is like picking a vehicle. A sports car wins every spec sheet, but if all you do is the weekly grocery run, a small hatchback is cheaper, easier to park, and gets you there just the same. With dozens of models to pick from, a clear decision framework beats a benchmark chart every time. Benchmarks tell you which model scores highest on standardized tests. They do not tell you which model works best for your task, your budget, and your latency limits. The model names in the code below are current at the time of writing, so swap in whatever each provider lists today; the logic of the framework does not change.

📄 model_selection.py: a decision framework for choosing an LLM

# Anvay builds a decision framework for his team
def recommend_llm(task, budget, latency, privacy):
    """Simple decision framework for LLM selection."""
    recommendations = []

    if privacy == "critical":
        recommendations.append(("Llama 3.1 (70B) via Ollama", "Full data privacy, no external API calls"))
        if latency == "low":
            recommendations.append(("Llama 3.2 (3B) quantized", "Fast local inference for simple tasks"))

    elif budget == "low":
        recommendations.append(("GPT-5.4 mini", "Most cost-effective tier, check openai.com/pricing"))
        recommendations.append(("Claude Haiku 4.5", "Fast and affordable, check anthropic.com/pricing"))
        if task in ("code", "reasoning"):
            recommendations.append(("DeepSeek V3", "Free tier available, strong at code"))

    elif task == "complex_reasoning":
        recommendations.append(("Claude Opus 4.8", "Strong extended thinking for hard problems"))
        recommendations.append(("GPT-5.5", "Strong all-rounder, multimodal"))

    elif task == "code":
        recommendations.append(("Claude Sonnet 4.6", "Strong coding model overall"))
        recommendations.append(("GPT-5.5", "Strong code generation, function calling"))

    elif task == "long_context":
        recommendations.append(("Claude Opus 4.8 (1M ctx)", "Largest context window from major providers"))
        recommendations.append(("Gemini 3.1 Pro", "Very large context window"))

    else:
        recommendations.append(("GPT-5.5", "Strong general-purpose model"))
        recommendations.append(("Claude Sonnet 4.6", "Strong alternative, good at nuance"))

    return recommendations

# Run some scenarios
scenarios = [
    {"task": "code", "budget": "medium", "latency": "medium", "privacy": "normal"},
    {"task": "summarization", "budget": "low", "latency": "low", "privacy": "normal"},
    {"task": "internal_docs", "budget": "any", "latency": "any", "privacy": "critical"},
    {"task": "complex_reasoning", "budget": "high", "latency": "high_ok", "privacy": "normal"},
]

for scenario in scenarios:
    recs = recommend_llm(**scenario)
    print(f"\nScenario: {scenario}")
    for model, reason in recs:
        print(f"  -> {model}: {reason}")

▶ Output

Scenario: {'task': 'code', 'budget': 'medium', 'latency': 'medium', 'privacy': 'normal'}
  -> Claude Sonnet 4.6: Strong coding model overall
  -> GPT-5.5: Strong code generation, function calling

Scenario: {'task': 'summarization', 'budget': 'low', 'latency': 'low', 'privacy': 'normal'}
  -> GPT-5.4 mini: Most cost-effective tier, check openai.com/pricing
  -> Claude Haiku 4.5: Fast and affordable, check anthropic.com/pricing

Scenario: {'task': 'internal_docs', 'budget': 'any', 'latency': 'any', 'privacy': 'critical'}
  -> Llama 3.1 (70B) via Ollama: Full data privacy, no external API calls

Scenario: {'task': 'complex_reasoning', 'budget': 'high', 'latency': 'high_ok', 'privacy': 'normal'}
  -> Claude Opus 4.8: Strong extended thinking for hard problems
  -> GPT-5.5: Strong all-rounder, multimodal

What happened here: We built a practical decision framework based on four real constraints: task type, budget, latency tolerance, and privacy requirements. In production, these constraints matter more than benchmark scores. A model that scores 2% higher on MMLU (Massive Multitask Language Understanding) but costs 10x more per token is rarely the right choice. The framework steers toward local models when privacy is critical and toward cheap API models when budget is the main constraint.

The Catch: LLMs Are Not Knowledge Databases

An LLM is closer to a supercharged autocomplete than to a librarian. The autocomplete on your phone never checks whether the word it suggests is true; it just predicts what usually comes next. The single biggest misconception about large language models is that they are search engines or databases. They are not. They are pattern completers. When an LLM “knows” that Paris is the capital of France, it does not look that fact up in a table.

It learned that the token “Paris” very often follows “The capital of France is” in its training data. That difference matters a lot, because it is exactly what explains hallucination. The model is always trying to write the most likely next words, not the most truthful ones, and most of the time likely and truthful happen to line up. Hallucination is what you get on the days they do not.

📄 hallucination_demo.py: why LLMs confidently say wrong things

# Understanding hallucination, the core LLM limitation
hallucination_types = {
    "Factual Hallucination": {
        "example": "LLM says 'Python was created in 1989 by Guido van Rossum'",
        "reality": "Python was first released in 1991 (conceived in late 1989)",
        "why": "Model saw both dates in training data, picked the wrong one",
        "risk": "Medium, verifiable facts can be checked",
    },
    "Citation Hallucination": {
        "example": "LLM cites 'Smith et al., 2023, Journal of ML Research'",
        "reality": "The paper does not exist",
        "why": "Model learned the PATTERN of citations, not actual papers",
        "risk": "High, looks authoritative but is fabricated",
    },
    "Reasoning Hallucination": {
        "example": "LLM solves a math problem with confident wrong steps",
        "reality": "Each step looks logical but the chain is broken",
        "why": "Model generates plausible next tokens, not verified logic",
        "risk": "Very high, hard to catch without doing the math yourself",
    },
    "Confident Nonsense": {
        "example": "LLM explains a fictional Python module 'pandas.quantum'",
        "reality": "No such module exists",
        "why": "Model interpolates between known patterns to create plausible fiction",
        "risk": "High for beginners who trust the output blindly",
    },
}

print("LLM Hallucination Types: Why Models Sound Confident When Wrong\n")
for htype, info in hallucination_types.items():
    print(f"  {htype}")
    print(f"    Example:  {info['example']}")
    print(f"    Reality:  {info['reality']}")
    print(f"    Why:      {info['why']}")
    print(f"    Risk:     {info['risk']}")
    print()

# Key defense strategies
defenses = [
    "RAG (Retrieval-Augmented Generation): ground answers in real documents",
    "Citation verification: always verify any references the LLM provides",
    "Temperature 0 for factual tasks: reduces creative interpolation",
    "Multi-model consensus: ask multiple models, compare answers",
    "Human-in-the-loop: critical decisions always reviewed by a person",
]

print("Defense strategies:")
for i, defense in enumerate(defenses, 1):
    print(f"  {i}. {defense}")

▶ Output

LLM Hallucination Types: Why Models Sound Confident When Wrong

  Factual Hallucination
    Example:  LLM says 'Python was created in 1989 by Guido van Rossum'
    Reality:  Python was first released in 1991 (conceived in late 1989)
    Why:      Model saw both dates in training data, picked the wrong one
    Risk:     Medium, verifiable facts can be checked

  Citation Hallucination
    Example:  LLM cites 'Smith et al., 2023, Journal of ML Research'
    Reality:  The paper does not exist
    Why:      Model learned the PATTERN of citations, not actual papers
    Risk:     High, looks authoritative but is fabricated

  Reasoning Hallucination
    Example:  LLM solves a math problem with confident wrong steps
    Reality:  Each step looks logical but the chain is broken
    Why:      Model generates plausible next tokens, not verified logic
    Risk:     Very high, hard to catch without doing the math yourself

  Confident Nonsense
    Example:  LLM explains a fictional Python module 'pandas.quantum'
    Reality:  No such module exists
    Why:      Model interpolates between known patterns to create plausible fiction
    Risk:     High for beginners who trust the output blindly

Defense strategies:
  1. RAG (Retrieval-Augmented Generation): ground answers in real documents
  2. Citation verification: always verify any references the LLM provides
  3. Temperature 0 for factual tasks: reduces creative interpolation
  4. Multi-model consensus: ask multiple models, compare answers
  5. Human-in-the-loop: critical decisions always reviewed by a person

What happened here: We sorted the four types of LLM hallucination by how risky each one is. The core issue is always the same: LLMs optimize for plausible text, not truthful text. A model that invents “Smith et al., 2023” is not lying on purpose. It is just producing the most statistically likely completion of a citation pattern.

This is exactly why retrieval-augmented generation (RAG), covered in the RAG tutorial, matters so much. Think of it as an open-book exam instead of answering from memory. First the system does retrieval, which simply means fetching the passages most relevant to your question from a document store (your own notes, a manual, a support knowledge base). Then it adds those passages into the prompt alongside your question, and the model generates an answer grounded in them and can cite exactly where each claim came from. Because the model is now reading from a real source instead of filling gaps from memory, RAG sharply cuts down on the hallucination you just saw.

Common Mistakes

Interview Corner

Q: What is the difference between a base model and an instruction-tuned model?

A base model (e.g., Llama 3.1 base) is trained only on next-token prediction. It completes text but does not follow instructions – ask it a question and it might continue writing the question instead of answering it. An instruction-tuned model (e.g., Llama 3.1 Instruct) has been further trained on instruction-response pairs (SFT) and aligned with human preferences (RLHF/DPO). The instruction-tuned version understands that when you ask a question, you want an answer.

Practice Exercises

1. Exercise 1: Use tiktoken with the gpt-4 encoding to count the tokens in a paragraph of your own writing, then in the same text translated to another language. Compare the token counts and explain why they differ.
2. Exercise 2: Extend the quantization table to add a Phi-3.5 Mini (3.8B) row and a 14B model row. Compute the memory at FP16 and INT4, then decide which of the four hardware tiers can run each one.
3. Exercise 3: Install Ollama, pull llama3.2:3b, and adapt the local-LLM snippet to ask the model the same factual question three times at temperature 0 and three times at temperature 1.0. Note which setting gives more consistent answers.

More in this series:

• NLP: Text Classification with HuggingFace Transformers
• GenAI: Prompt Engineering, Techniques and Best Practices
• GenAI: LLM APIs (OpenAI, Anthropic, Structured Output & Function Calling)

Frequently Asked Questions

Interview Questions on Large Language Models

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: What are emergent abilities, and why do they matter when choosing a model?

Emergent abilities are capabilities that show up only after a model crosses a certain scale threshold, such as chain-of-thought reasoning or reliable few-shot learning. A small model simply cannot do them, and then a larger one can, with no change to the architecture. They matter because you cannot always predict a bigger model’s behavior from a smaller one, so a task that fails on a 7B model may just work on a 70B model. One honest caveat: researchers still debate how sharp these jumps are, partly because of the all-or-nothing way the tasks are scored.

Q: Why does quantization let a 70B model run on a single GPU, and what is the cost?

Quantization stores each parameter in fewer bits, for example INT4 instead of FP32, which cuts memory by roughly 8x. A 70B model drops from about 260 GB at full precision to about 33 GB at 4-bit, which fits on one high-end GPU. The cost is a small accuracy loss, typically 1 to 3 percent on benchmarks, which is usually a fine trade for local or prototyping work but may matter for high-stakes accuracy.

Q: What are the three stages of LLM training, and what does each one add?

Pre-training uses next-token prediction over trillions of tokens to build raw language ability and world knowledge, but the result is only a text-completion engine. Supervised fine-tuning (SFT) on instruction-response pairs teaches the model to actually follow instructions. Alignment (RLHF, DPO, or Constitutional AI) tunes it toward human preferences and safety. That layered pipeline is why a base model and its chat variant behave so differently.

Q: What is a context window, and why can a larger one still produce worse answers?

The context window is the maximum number of tokens the model can attend to at once, effectively its short-term memory. Bigger windows let you pass whole documents, but models often struggle with information buried in the middle of a very long prompt, the “lost in the middle” effect. Put the most relevant context near the beginning and end, and do not pad the prompt just because the window allows it.

Q: Your LLM-powered support bot confidently cites a refund policy clause that does not exist. What is happening and how do you fix it?

That is a hallucination: the model produced the most statistically likely completion of a policy-sounding pattern, not a retrieved fact. The fix is retrieval-augmented generation (RAG) so answers are grounded in your actual policy documents, plus a lower temperature for factual queries and a citation-verification step. Never let raw model output drive a policy statement without grounding and human review.

Q: Your API bill for a summarization feature spikes 10x after you route every request to a flagship model. What do you check first?

First check whether the task even needs a flagship model. Summarization, classification, and extraction usually run just as well on a cheaper mini or small model, so route only genuinely hard requests to the flagship and default the rest to a smaller tier. Also confirm you are not sending an unnecessarily large context on every call, since input tokens are billed too and a bloated prompt multiplies the cost.

Q: A healthcare client requires that no patient data ever leaves their servers. Which LLM approach fits, and what do you trade away?

Use an open-weight model such as Llama running locally through Ollama or a self-hosted server, so nothing is sent to an external API. Quantize it to fit the available GPU memory. You trade some peak capability versus a frontier closed model, but you gain full data privacy and zero per-token cost, which is exactly what the compliance requirement demands.

What’s Next?

You now understand what large language models are: the same Transformer scaled up until new abilities emerge, trained in three stages (pre-training, SFT, alignment), limited by context windows and quantization trade-offs, and prone to hallucination because they complete patterns rather than look up facts. You also have a first feel for choosing one. The next post, the how to choose an LLM tutorial, turns that instinct into a repeatable decision framework: open versus closed, model size, cost, and licenses, so you can pick the right model for a project and defend the choice to your team.

Want the whole journey from Python basics to production AI in order? Keep the Python + AI/ML tutorial series home bookmarked as your map.

Go deeper: when you outgrow this post, Hugging Face documentation is the next stop.

Related Posts

Previous: NLP: Text Classification with HuggingFace Transformers

Next: How LLMs Are Trained: Pretraining, SFT, and RLHF Explained

Series Home: Python + AI/ML Tutorial Series