A large language model is never sat down and taught facts. It learns by predicting the next token across a mountain of text, then gets reshaped twice more until it behaves like the assistant you talk to. That is how LLMs are trained: pretraining, supervised fine-tuning, and preference tuning. This post walks all three stages in plain words, with small Python you can run yourself.

“The computer was born to solve problems that did not exist before.”

Bill Gates

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Intermediate | Reading Time: 20 minutes

In the LLM introduction you saw what these models can do and how big they are. This is the companion piece that answers the natural next question: where does that ability come from? You do not need a datacenter to understand the recipe. By the end you will be able to explain, to a friend or in an interview, exactly what pretraining, SFT, and RLHF each contribute and why the order matters.

The Three Stages, in Plain Words

Think of training an assistant like raising a very well-read intern. First they read almost everything ever written and get an uncanny feel for how sentences tend to continue. That is pretraining, and it produces a base model that can autocomplete anything but does not know it is supposed to answer your questions. Next you hand the intern a stack of worked examples that show the format you want: here is a question, here is a good answer.

That is supervised fine-tuning, and it turns the base model into an instruct model that follows directions. Finally you let people compare answers and mark which they prefer, and you nudge the model toward the preferred style. That is preference tuning, and it produces the polished assistant.

Each stage uses a different kind of data and a different amount of it. Pretraining swallows trillions of tokens of raw text. SFT uses maybe a hundred thousand carefully written pairs. Preference tuning uses comparisons between answers. The diagram below is the whole assembly line on one screen.

Notice the shape: each stage takes the model from the stage before it and adds one new ingredient. Pretraining needs no labels at all, which is why it can use the whole web. The later stages need human-written or human-judged data, which is expensive, so they are small by comparison. That single fact explains most of what follows.

Stage 1: Pretraining, Predicting the Next Token

The engine under every LLM is embarrassingly simple to state: given some text, predict the next token. Do that a few trillion times, adjusting the model each time it guesses wrong, and it slowly builds an internal sense of grammar, facts, and reasoning patterns. Nobody labels the data. The next word in the sentence is its own label, which is why this is called self-supervised learning.

You can feel the whole idea with a toy model that has none of the neural-network machinery. Picture how you finish a friend’s sentence: you have heard “for” followed by “lunch” or “dinner” so many times that you can rank the likely next words. The script below counts which word follows which in a tiny corpus, then uses those counts to predict and even to generate. It is a bigram model, the great-great-grandparent of a modern LLM, but the loop is the same one: look at context, rank the next token, pick one, repeat.

📄 ngram_toy.py: next-token prediction with pure counting

"""A tiny next-token predictor, the same core idea a giant LLM learns, in ~25 lines."""
from collections import defaultdict, Counter

# A tiny "training corpus". A real LLM reads trillions of tokens; we use a handful of lines.
corpus = """
aditi cooks paneer for dinner
aditi cooks rice for lunch
anvay cooks paneer for lunch
anvay orders pizza for dinner
aviraj cooks rice for dinner
aditi orders pizza for lunch
anvay cooks paneer for dinner
""".split()

# Learn: for each word, count which word tends to follow it (a bigram model).
counts = defaultdict(Counter)
for a, b in zip(corpus, corpus[1:]):
    counts[a][b] += 1

def next_token_probs(word):
    followers = counts[word]
    total = sum(followers.values())
    return [(tok, n / total) for tok, n in followers.most_common()]

for prompt in ["cooks", "for", "aditi"]:
    print(f"After '{prompt}', the model expects:")
    for tok, p in next_token_probs(prompt):
        print(f"    {tok:8} {p:.2f}")
    print()

# Generate text by predicting one token at a time and feeding it back in (greedy).
word, out = "aditi", ["aditi"]
for _ in range(4):
    probs = next_token_probs(word)
    if not probs:
        break
    word = probs[0][0]          # pick the single most likely next token
    out.append(word)
print("Generated:", " ".join(out))

▶ Output

After 'cooks', the model expects:
    paneer   0.60
    rice     0.40

After 'for', the model expects:
    dinner   0.57
    lunch    0.43

After 'aditi', the model expects:
    cooks    0.67
    orders   0.33

Generated: aditi cooks paneer for dinner

What happened here: The model never stored a single fact about food. It only counted co-occurrences, yet it now “knows” that cooks is followed by paneer more often than rice (0.60 versus 0.40), and it can spin out a plausible sentence by greedily taking the top pick each step. A real LLM does the same thing with two upgrades: it looks back over thousands of tokens instead of one, and it learns a smooth function instead of a lookup table, so it can predict sensibly for contexts it has never seen exactly. That single generalization is where grammar, style, and a surprising amount of world knowledge quietly come from.

Tokenization: How Text Becomes Tokens

One detail we skipped: models do not predict words, they predict tokens. A token is a chunk of text, often a whole common word but sometimes a piece of one. Before pretraining can start, the training team learns a vocabulary of tokens from the data itself, using an algorithm called Byte Pair Encoding (BPE). The idea is childlike: start with single characters, then repeatedly glue together the most common neighbouring pair into a new token. Frequent stuff like cook becomes one token, rare stuff gets spelled out from pieces.

Here is BPE built from scratch so you can watch the vocabulary form. Think of it like learning shorthand: the squiggles you write most often earn their own single symbol, and everything else you spell out.

📄 bpe_toy.py: learning a subword vocabulary the way real tokenizers do

"""Byte Pair Encoding (BPE) from scratch: how LLMs learn their subword vocabulary."""
from collections import Counter

# A tiny corpus. Real tokenizers learn from billions of words; the algorithm is identical.
words = ("cooking cooked cooks booking booked books "
         "looking looked looks").split()

# Start every word as a list of single characters, plus a "_" end-of-word marker.
vocab = {tuple(w) + ("_",): 1 for w in words}

def pair_counts(vocab):
    pairs = Counter()
    for symbols, freq in vocab.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(vocab, pair):
    a, b = pair
    new_vocab = {}
    for symbols, freq in vocab.items():
        merged, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == (a, b):
                merged.append(a + b)   # glue the pair into one new token
                i += 2
            else:
                merged.append(symbols[i])
                i += 1
        new_vocab[tuple(merged)] = freq
    return new_vocab

# Learn the 6 most useful merges, one at a time, always merging the most frequent pair.
merges = []
for step in range(6):
    pairs = pair_counts(vocab)
    best = pairs.most_common(1)[0][0]
    merges.append(best)
    vocab = merge(vocab, best)
    print(f"Merge {step + 1}: joined {best[0]!r} + {best[1]!r} -> {best[0] + best[1]!r}")

# Apply the learned merges to a word, including one the tokenizer never saw in training.
def tokenize(word):
    symbols = list(word) + ["_"]
    for a, b in merges:
        i = 0
        while i < len(symbols) - 1:
            if (symbols[i], symbols[i + 1]) == (a, b):
                symbols[i:i + 2] = [a + b]
            else:
                i += 1
    return symbols

print()
for w in ["cooking", "cookbook"]:
    print(f"{w:9} -> {tokenize(w)}")

▶ Output

Merge 1: joined 'o' + 'o' -> 'oo'
Merge 2: joined 'oo' + 'k' -> 'ook'
Merge 3: joined 'c' + 'ook' -> 'cook'
Merge 4: joined 'i' + 'n' -> 'in'
Merge 5: joined 'in' + 'g' -> 'ing'
Merge 6: joined 'ing' + '_' -> 'ing_'

cooking   -> ['cook', 'ing_']
cookbook  -> ['cook', 'b', 'ook', '_']

What happened here: From nothing but frequency counts, BPE discovered that cook, ook, and ing deserve to be tokens. So a common word like cooking becomes just two tokens, while cookbook, which the tokenizer never saw during training, still gets handled gracefully by falling back to pieces it does know: cook, b, ook. This is why an LLM never truly hits an unknown word. Worst case it spells one out character by character. It is also why token count, not word count, is what you pay for on an Application Programming Interface (API) and what fills a context window, the fixed number of tokens a model can take in at once.

Why Base Models Are Weird

Straight out of pretraining you have a base model, and talking to one is a strange experience. Ask it “What is the capital of France?” and instead of answering it might continue with “What is the capital of Germany? What is the capital of Italy?” because in its training data, questions like that often appeared in lists. The base model is not broken. It is doing its one job perfectly: predicting a plausible continuation. It just was never told that when a human types a question, the human wants an answer, not more questions.

Imagine hiring someone who has read every book in the library but has never had a conversation. They know an enormous amount, yet if you ask them something they might just recite more text on the topic rather than reply to you. That gap between “knows a lot” and “is helpful” is exactly what the next two stages close. The knowledge is already in the base model. What is missing is the behaviour.

Stage 2: Supervised Fine-Tuning (SFT)

Supervised fine-tuning teaches the base model the format of being helpful. You collect a set of example conversations written by humans, each one an instruction paired with a good response, and you continue training on them with the same next-token objective as before. Because the examples always follow the pattern “user asks, assistant answers well”, the model learns to fall into that pattern whenever it sees a user turn. It is the same trick from the HuggingFace fine-tuning tutorial, just aimed at conversational behaviour instead of a single classification label.

The data is the whole game here. Each example is wrapped in a fixed chat template so the model can tell the user’s turn from the assistant’s turn. The little script below shows what one training example actually looks like after formatting.

📄 sft_format.py: turning instruction pairs into the text SFT trains on

"""SFT teaches format: turn plain (instruction, response) pairs into chat-template text."""

# A supervised fine-tuning set is thousands of hand-written pairs like these.
pairs = [
    {"instruction": "Suggest a quick vegetarian lunch.",
     "response": "Try a paneer wrap with salad. It takes about ten minutes."},
    {"instruction": "What is 12 times 8?",
     "response": "12 times 8 is 96."},
]

# During SFT the model sees each pair wrapped in a fixed chat template and learns to
# produce the assistant turn given the user turn.
def to_chat(pair):
    return (f"<|user|>\n{pair['instruction']}\n"
            f"<|assistant|>\n{pair['response']}<|end|>")

for p in pairs:
    print(to_chat(p))
    print("-" * 40)

▶ Output

<|user|>
Suggest a quick vegetarian lunch.
<|assistant|>
Try a paneer wrap with salad. It takes about ten minutes.<|end|>
----------------------------------------
<|user|>
What is 12 times 8?
<|assistant|>
12 times 8 is 96.<|end|>
----------------------------------------

What happened here: The special markers like <|user|> and <|assistant|> are just tokens that tell the model whose turn it is. During SFT the model is trained to predict the assistant’s text after seeing the user’s text and the assistant marker. After a few thousand examples like this, the base model stops autocompleting lists of questions and starts answering them. It has not learned any new facts, only a new habit. That is why SFT is quick and cheap compared to pretraining: you are steering an existing model, not building one.

Stage 3: Preference Tuning (RLHF and DPO)

After SFT the model is helpful but not yet polished. It might answer correctly while being rude, rambling, or unsafe. Preference tuning fixes the quality and values of the answers. The setup is intuitive: show people two responses to the same prompt and ask which they prefer. Collect a pile of these “chosen versus rejected” judgements, then adjust the model to make preferred answers more likely and rejected ones less likely.

Say a user named Anvi asks for a recipe. Two answers come back: one is a clear numbered list, the other is a wall of text with a missing step. Anvi picks the first. Multiply that choice by millions and you have taught the model what “good” looks like to people, without ever writing down an explicit rule for it. This is where a model learns to refuse harmful requests, admit uncertainty, and keep a friendly tone.

Two methods dominate at the time of writing (mid-2026). RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model to score answers, then uses reinforcement learning to push the LLM toward high-scoring responses. DPO (Direct Preference Optimization) skips the separate reward model and optimizes the preference data directly, which is simpler and has become popular for open-weight models. Both aim at the same target: make the model prefer what people prefer. The mechanics of each get their own hands-on treatment in the RLHF versus DPO deep dive; here the thing to hold onto is why the stage exists, not the math.

Where Open-Weight Models Come From

Every model you download and run yourself came off an assembly line like this one, paid for by an organization with a very large compute budget. At the time of writing (mid-2026), the well-known open-weight families include Meta’s Llama, Alibaba’s Qwen, Mistral’s models, Google’s Gemma, and Microsoft’s Phi. These labs run all three stages, then release the trained weights so anyone can download them from a hub like HuggingFace and run or fine-tune them for free.

It helps to separate two words that get muddled. “Open-weight” means the finished parameters are published, so you can run the model, but the training data and code often are not. “Open-source” in the strict sense would mean the data and recipe are public too, which is rarer. For a working developer the practical point is the same: you almost never pretrain. You start from someone else’s base or instruct model and do the cheap stages yourself. The specific model names above will rotate over time, so treat them as dated examples and check the current leaderboards when you choose, but the release pattern (a lab pretrains, then shares weights) has held steady across every model generation so far.

The Scale and Cost: Why Nobody Pretrains at Home

Here is the concrete reason pretraining lives in datacenters. There is a well-worn rule of thumb that the compute needed to train a model is roughly 6 × parameters × tokens floating-point operations. Plug in real numbers and the gap between a toy model and a frontier model becomes physical, not abstract.

📄 pretrain_cost.py: estimating pretraining compute in GPU-hours

"""Back-of-envelope: why nobody pretrains a frontier model on a home GPU."""

# A well-known rule of thumb: training compute in FLOPs is about 6 * params * tokens.
def pretrain_flops(params, tokens):
    return 6 * params * tokens

# One modern datacenter GPU does very roughly 1e15 useful FLOPs per second (with real-world
# utilization well under the sticker peak). A year has about 3.15e7 seconds.
GPU_FLOPS_PER_SEC = 1e15

for name, params, tokens in [
    ("Small (1B params, 20B tokens)", 1e9, 20e9),
    ("Mid (8B params, 8T tokens)", 8e9, 8e12),
    ("Frontier (70B params, 15T tokens)", 70e9, 15e12),
]:
    flops = pretrain_flops(params, tokens)
    gpu_hours = flops / GPU_FLOPS_PER_SEC / 3600
    # If you rent 1,000 GPUs at once, wall-clock time is gpu_hours / 1000.
    wall_days = gpu_hours / 1000 / 24
    print(f"{name}")
    print(f"    total compute : {flops:.1e} FLOPs")
    print(f"    one GPU        : {gpu_hours:,.0f} GPU-hours")
    print(f"    on 1,000 GPUs  : ~{wall_days:.2f} wall-clock days")
    print()

▶ Output

Small (1B params, 20B tokens)
    total compute : 1.2e+20 FLOPs
    one GPU        : 33 GPU-hours
    on 1,000 GPUs  : ~0.00 wall-clock days

Mid (8B params, 8T tokens)
    total compute : 3.8e+23 FLOPs
    one GPU        : 106,667 GPU-hours
    on 1,000 GPUs  : ~4.44 wall-clock days

Frontier (70B params, 15T tokens)
    total compute : 6.3e+24 FLOPs
    one GPU        : 1,750,000 GPU-hours
    on 1,000 GPUs  : ~72.92 wall-clock days

What happened here: A 1-billion-parameter toy needs about 33 Graphics Processing Unit (GPU)-hours, which is genuinely a weekend on a single rented card. A frontier-scale run needs around 1.75 million GPU-hours, which even on a thousand GPUs at once still takes over two months of wall-clock time, before you count failed runs, data cleaning, and salaries. That is why pretraining is done by a handful of labs and everyone else starts from released weights. This ties straight back to the hardware guide: the cheapest tool that finishes your job is almost never a pretraining cluster. Fine-tuning an existing model, which you will do in the fine-tuning tutorial, costs a rounding error by comparison.

Common Mistakes

Best Practices

Conclusion

So that is how LLMs are trained, start to finish. Pretraining reads the web and learns to predict the next token, which quietly builds knowledge and language skill into a base model. Supervised fine-tuning teaches that base model the format of being helpful and turns it into an instruct model. Preference tuning, whether through RLHF or DPO, polishes the answers toward what people actually want. Each stage stacks on the one before, needs less data than the one before, and the whole thing has stayed structurally the same across every model generation so far, even as the specific models and tools keep changing.

The practical takeaway is freeing: you never have to run stage one. The expensive part is already done and shared. Your job is to pick a good released model and, when you need to, do the cheap stages yourself. Next up, put that into practice in the LLM fine-tuning tutorial. For the full path from Python basics to production AI, head to the Python + AI/ML Cookbook tutorial series home.

Frequently Asked Questions

Interview Questions on How LLMs Are Trained

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: Walk me through the three stages of training a modern LLM.

Pretraining comes first: the model learns to predict the next token on trillions of tokens of unlabeled text, which builds language skill and knowledge into a base model. Supervised fine-tuning comes next: training on human-written instruction and response pairs teaches the model to follow directions, producing an instruct model. Preference tuning is last: using human comparisons of answers, via RLHF or DPO, the model is aligned toward helpful, honest, and safe responses. Each stage reuses the model from the previous one and uses progressively less but higher-quality data.

Q: Why is pretraining called self-supervised rather than unsupervised?

Because there is a label for every training example, but it comes from the data itself rather than a human annotator. The label is simply the next token in the sequence. The model makes a prediction, compares it to the actual next token, and updates. There is supervision, just no manual labelling, which is what lets pretraining scale to the entire web.

Q: A base model answers a question by generating more questions. What is going on and how do you fix it?

The base model is doing exactly what pretraining trained it to do: predict a plausible continuation. In its data, a question is often followed by more questions, so it continues the pattern instead of answering. The fix is supervised fine-tuning on instruction and response pairs, which teaches the model that a user turn should be followed by a helpful assistant turn. In practice you just use the instruct version of the model.

Q: Does RLHF add knowledge to a model?

No. Preference tuning changes behaviour, style, safety, and helpfulness, not the underlying knowledge, which is set during pretraining. If the base model never learned a fact, no amount of preference tuning will insert it. To add or update knowledge you use retrieval-augmented generation or fine-tuning on new content, not alignment.

Q: Why does subword tokenization with BPE matter for training?

BPE gives the model a fixed, manageable vocabulary while still handling any word. Common words become single tokens for efficiency, and rare or unseen words are split into known subword pieces, so the model never faces a truly unknown input. This keeps the vocabulary small enough to train on, controls sequence length, and is why cost and context limits are measured in tokens rather than words.

Q: Roughly how much compute does pretraining take, and why does that shape the industry?

Using the rule of thumb that compute is about six times parameters times tokens, a frontier model runs into the order of a million-plus GPU-hours. That is months of wall-clock time even on a thousand GPUs, so only well-funded labs can pretrain. Everyone else starts from released weights and fine-tunes, which is why open-weight releases and the fine-tuning ecosystem matter so much.

Q: Scenario: a teammate named Aviraj wants to give your assistant knowledge of your internal docs. Should he pretrain, do SFT, or something else?

Not pretraining, which is far too expensive, and SFT is the wrong tool for adding facts since it teaches behaviour rather than knowledge. The right first move is retrieval-augmented generation: index the internal docs and feed the relevant chunks into the prompt at query time. If the docs are stable and he needs the model to internalize a style or format, light fine-tuning on top of an instruct model is a reasonable second step. Start with retrieval because it is cheapest and easiest to keep current.

More in this series:

• NLP: BERT, GPT, and Modern Language Models
• How to Choose an LLM: Open vs Closed, Size, Cost, Licenses
• GenAI: Prompt Engineering, Techniques and Best Practices

Background: GenAI: Introduction to Large Language Models

Next up: GenAI: Fine-Tuning LLMs for Your Own Task

Series Home: Python + AI/ML Cookbook. Complete Tutorial Series

Further reading: for the full reference, see Hugging Face documentation.

Related Posts

Previous: GenAI: Introduction to Large Language Models

Next: How to Choose an LLM: Open vs Closed, Size, Cost, Licenses

Series Home: Python + AI/ML Tutorial Series