Supervised fine-tuning gets you a model that answers correctly yet still manages to be tactless, wordy, or too eager to please. The last polish comes from preference tuning, and RLHF vs DPO is the choice of route: one trains a reward model and runs reinforcement learning, the other folds the whole objective into a single loss. This post explains both in plain words and runs the DPO loss for real in Python.

“A model does not learn what you tell it is good. It learns what you keep picking when shown two answers side by side.”

Last Updated: July 2026 | Tested on: Python 3.14.6, PyTorch 2.12.1 | Difficulty: Advanced | Reading Time: 22 minutes

In the three-stage training walkthrough you saw that preference tuning is the final stage that turns a capable instruct model into a polished assistant. That post named RLHF and DPO and promised a closer look. This is it. By the end you will be able to explain, in an interview or to a teammate, why supervised fine-tuning alone leaves a gap, how RLHF closes it with a reward model, why DPO closes the same gap without one, and which trade-offs push a team toward each.

Why SFT Is Not Enough

Supervised fine-tuning trains a model to copy one good answer per prompt. That teaches format and manners, but it has no way to say “this answer is fine, and this other one is better”. Think of teaching someone to cook by only ever showing them the finished dish. They can imitate it, but they never learn why one plating beats another, so their taste never sharpens. Preference tuning fixes exactly that: instead of one target answer, you show the model two answers and tell it which one a human preferred.

The gap matters because most of what we want from an assistant is a matter of degree, not correctness. Two answers can both be true, on-topic, and grammatical, while one is crisp and the other is a vague ramble. SFT cannot express “both are valid, prefer the crisp one”. Preference data can.

Preference Pairs: The Data Alignment Runs On

The raw material for both RLHF and DPO is the same: a pile of preference pairs. Each one is a prompt, a chosen answer, and a rejected answer, where a human decided the chosen one was better. Say a reviewer named Aditi is shown two dinner suggestions for the same request and clicks the clearer one. That single click is one training example. The little script below puts an SFT example next to a preference pair so you can see what extra signal the pair carries.

📄 preference_pairs.py: what preference data encodes that SFT cannot

"""Why SFT is not enough: SFT has one 'right' answer, preference data ranks two."""

# Supervised fine-tuning (SFT) data: one prompt, one gold answer. That is all it can say.
sft_example = {
    "prompt": "Suggest a quick vegetarian dinner.",
    "answer": "Make a paneer stir-fry with rice. It is ready in about fifteen minutes.",
}

# Preference data: same prompt, but now TWO answers ranked by a human. This encodes
# taste, not just correctness. Both answers are 'right'; one is simply better.
preference_pair = {
    "prompt": "Suggest a quick vegetarian dinner.",
    "chosen":   "Try a paneer stir-fry: saute paneer and veggies, add soy sauce, "
                "serve over rice. About 15 minutes.",
    "rejected": "You could make dinner using vegetables and some protein source of "
                "your choosing, prepared in whatever way you prefer.",
}

print("SFT sees ONE target answer:")
print("   ", sft_example["answer"])
print()
print("Preference data ranks TWO answers for the SAME prompt:")
print("   chosen  :", preference_pair["chosen"][:60], "...")
print("   rejected:", preference_pair["rejected"][:60], "...")
print()

# The signal SFT cannot express: 'the rejected answer is also valid English and also
# on-topic, it is just vaguer'. Count what each format teaches the model.
print("What each format teaches:")
print(f"   SFT        -> imitate 1 answer  (no notion of 'better')")
print(f"   preference -> chosen > rejected (a direction to move in)")

▶ Output

SFT sees ONE target answer:
    Make a paneer stir-fry with rice. It is ready in about fifteen minutes.

Preference data ranks TWO answers for the SAME prompt:
   chosen  : Try a paneer stir-fry: saute paneer and veggies, add soy sau ...
   rejected: You could make dinner using vegetables and some protein sour ...

What each format teaches:
   SFT        -> imitate 1 answer  (no notion of 'better')
   preference -> chosen > rejected (a direction to move in)

What happened here: Both answers in the pair are perfectly acceptable English and both address the request. The only new information is the ranking: a human said the specific, step-by-step answer beats the vague one. SFT has no slot for that comparison; it can only be handed a single gold answer to imitate. Preference tuning turns that ranking into a direction the model can move in, which is the whole reason this final stage exists. The rest of the post is about the two ways to actually apply that direction.

The Two Roads From Preference Data

Once you have preference pairs, you have to turn “chosen beats rejected” into weight updates. There are two established roads, and that fork is the whole RLHF vs DPO question. RLHF trains a separate reward model that learns to score any answer, then uses reinforcement learning to push the language model toward high-scoring answers. DPO throws out the separate reward model and derives a single loss that raises the chosen answer’s probability and lowers the rejected one’s directly. The diagram lays both routes side by side so you can see where they share the map and where they split.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

Notice the shape. RLHF is a loop: the model generates, the reward model scores, an RL step nudges the weights, and a leash back to a frozen reference model stops it drifting. DPO is a straight line: feed the pairs into one loss and step. Same starting data, same destination, very different amount of moving machinery. That difference in machinery is most of the practical RLHF vs DPO story.

RLHF: A Reward Model in the Loop

RLHF stands for Reinforcement Learning from Human Feedback, and it works in two moves. First you train a reward model: show it the preference pairs and teach it to output a higher number for the chosen answer than the rejected one. Now you have an automatic judge that can score any answer, not just the ones humans looked at. Second, you let the language model (the policy) generate answers, have the reward model score them, and use reinforcement learning to push the policy toward higher scores. It is like a chef cooking dish after dish while a critic rates each plate, and the chef adjusts to earn better ratings.

There is a catch that gives RLHF its trickiest part. The reward model is imperfect, so if you let the policy chase reward with no restraint, it finds cheap tricks that fool the judge instead of genuinely improving. This is called reward hacking: padded, sycophantic, or oddly formatted answers that the reward model happens to over-rate. The fix is a leash, a KL-divergence penalty that punishes the policy for straying too far from the original reference model. The script below is a tiny numeric stand-in for that whole tension: four candidate answers, a reward for each, and a “drift” that plays the role of KL distance from the reference.

📄 rlhf_kl.py: reward hacking and why the KL leash exists

"""RLHF in miniature: reward model + KL leash. Shows why the leash exists."""
import math

# Imagine 4 candidate answers to one prompt. In real RLHF a learned reward model
# scores each one. Here we hand-write the scores. Notice answer D games the reward:
# it is padded and sycophantic, so the reward model (imperfect!) over-rates it.
candidates = {
    "A concise correct answer":      {"reward": 0.80, "drift": 0.10},
    "A slightly longer good answer": {"reward": 0.85, "drift": 0.30},
    "A rambling but ok answer":      {"reward": 0.60, "drift": 0.90},
    "A padded sycophantic answer":   {"reward": 0.95, "drift": 2.50},  # reward hack
}

# 'drift' stands in for KL divergence from the SFT reference model: how far this
# answer pulls the policy away from where it started. RLHF maximizes:
#     objective = reward - beta * KL
# beta is the strength of the leash back to the reference model.
def best_answer(beta):
    scored = {name: v["reward"] - beta * v["drift"] for name, v in candidates.items()}
    winner = max(scored, key=scored.get)
    return winner, scored

for beta in [0.0, 0.1, 0.3]:
    winner, scored = best_answer(beta)
    print(f"beta = {beta}  (leash strength)")
    for name, s in scored.items():
        mark = "  <-- picked" if name == winner else ""
        print(f"    {s:+.2f}  {name}{mark}")
    print()

▶ Output

beta = 0.0  (leash strength)
    +0.80  A concise correct answer
    +0.85  A slightly longer good answer
    +0.60  A rambling but ok answer
    +0.95  A padded sycophantic answer  <-- picked

beta = 0.1  (leash strength)
    +0.79  A concise correct answer
    +0.82  A slightly longer good answer  <-- picked
    +0.51  A rambling but ok answer
    +0.70  A padded sycophantic answer

beta = 0.3  (leash strength)
    +0.77  A concise correct answer  <-- picked
    +0.76  A slightly longer good answer
    +0.33  A rambling but ok answer
    +0.20  A padded sycophantic answer

What happened here: With beta = 0.0 there is no leash, so the policy happily picks the padded sycophantic answer because the imperfect reward model scored it highest. That is reward hacking in one line. Turn the leash up to 0.1 and the hack gets penalized for drifting far from the reference, so a genuinely good answer wins. Push it to 0.3 and the model plays it safest of all, staying closest to where SFT left it. Real RLHF is this same tug of war at massive scale: reward pulls the model to improve, KL holds it back from cheating, and tuning beta is the art of the whole method.

DPO: Skipping the Reward Model

DPO stands for Direct Preference Optimization, and its big idea is that you do not need a separate reward model or a reinforcement learning loop at all. The DPO authors showed with some algebra that the reward-model-plus-RL objective can be rewritten as a single loss you can train with plain gradient descent, exactly like ordinary supervised learning. The reference model is still there, keeping the same KL leash, but it stays frozen and you never train a reward model or sample new answers during training. Fewer moving parts, fewer things to break, which is why DPO became the default for open-weight preference tuning.

The loss for one pair is short. It takes the policy’s log-probability of the chosen and rejected answers, subtracts the frozen reference model’s log-probabilities of the same two, and pushes the gap in the chosen direction. Rather than hand-wave the formula, the script below runs it for real in PyTorch on toy log-probabilities. No model download, no Graphics Processing Unit (GPU): just five preference pairs and the actual DPO loss, trained for sixty steps so you can watch it work.

📄 dpo_loss.py: the real DPO objective on toy log-probs

"""The DPO objective, run for real in PyTorch on toy log-probs (no model download).

DPO replaces RLHF's reward model with one loss. For a (prompt, chosen, rejected)
triple it pushes the policy to raise the log-prob of 'chosen' and lower 'rejected',
while a frozen reference model keeps it from wandering off (the same KL idea as RLHF,
folded into the loss). The loss for one pair is:

    loss = -log_sigmoid( beta * ( (lp_chosen  - ref_chosen)
                                 -(lp_rejected - ref_rejected) ) )
"""
import torch
import torch.nn.functional as F

torch.manual_seed(0)
beta = 0.1

# Frozen reference model's log-probs for 5 pairs (chosen, rejected). These never change.
ref_chosen   = torch.tensor([-2.0, -2.5, -1.8, -3.0, -2.2])
ref_rejected = torch.tensor([-2.1, -2.4, -2.0, -2.8, -2.3])

# The policy starts as an exact copy of the reference, so at step 0 it has no opinion.
policy_chosen   = ref_chosen.clone().requires_grad_(True)
policy_rejected = ref_rejected.clone().requires_grad_(True)

opt = torch.optim.SGD([policy_chosen, policy_rejected], lr=0.5)

def dpo_loss():
    chosen_logratio   = policy_chosen   - ref_chosen
    rejected_logratio = policy_rejected - ref_rejected
    margin = chosen_logratio - rejected_logratio
    return -F.logsigmoid(beta * margin).mean()

print(f"{'step':>4} | {'loss':>7} | {'chosen reward':>13} | {'rejected reward':>15}")
for step in range(0, 61):
    loss = dpo_loss()
    if step % 15 == 0:
        # 'Implicit reward' in DPO is beta * (policy_logprob - ref_logprob), averaged.
        r_chosen   = (beta * (policy_chosen   - ref_chosen)).mean().item()
        r_rejected = (beta * (policy_rejected - ref_rejected)).mean().item()
        print(f"{step:>4} | {loss.item():>7.4f} | {r_chosen:>13.4f} | {r_rejected:>15.4f}")
    opt.zero_grad()
    loss.backward()
    opt.step()

# Did the model end up preferring 'chosen' on every pair?
final_margin = (policy_chosen - ref_chosen) - (policy_rejected - ref_rejected)
print()
print("final per-pair margin (chosen - rejected, want > 0):")
print("   ", [round(x, 3) for x in final_margin.tolist()])
print("   all positive:", bool((final_margin > 0).all()))

▶ Output

step |    loss | chosen reward | rejected reward
   0 |  0.6931 |        0.0000 |          0.0000
  15 |  0.6857 |        0.0075 |         -0.0075
  30 |  0.6784 |        0.0149 |         -0.0149
  45 |  0.6711 |        0.0223 |         -0.0223
  60 |  0.6640 |        0.0296 |         -0.0296

final per-pair margin (chosen - rejected, want > 0):
    [0.601, 0.601, 0.601, 0.601, 0.601]
   all positive: True

What happened here: At step 0 the policy is an exact copy of the reference, so it has no preference and the loss is 0.6931, which is just the natural log of 2, the loss of a coin flip. As training runs, the chosen reward climbs above zero and the rejected reward sinks below it by the same amount, which is the loss doing precisely its job: raise chosen, lower rejected.

By the end every pair has a positive margin, meaning the model now prefers the chosen answer over the rejected one on all five. That “implicit reward” you see printed is the clever part of DPO: it never trained a reward model, yet beta × (policy log-prob minus reference log-prob) behaves exactly like one. The reward model did not disappear, it got folded into the loss.

Hands-On: One DPO Step With TRL

The toy loss above is the whole engine, and it is also where RLHF vs DPO stops being abstract: in practice you never hand-code it. The standard tool at the time of writing is Hugging Face TRL, whose DPOTrainer wraps a real model, builds the frozen reference for you, computes the log-probabilities, and runs the exact loss you just saw. The code below fine-tunes a small instruct model on fifty toy preference pairs. It needs a model download and a GPU to run comfortably, so treat the output as a representative run rather than something this article produced on a stdlib box.

📄 dpo_trl.py: a 50-pair DPO step with Hugging Face TRL

"""A minimal DPO run with TRL. Needs a GPU and a model download; toy scale for teaching."""
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

# A small instruct model keeps this runnable on a single modest GPU (Colab-sized).
# The exact model will age; pick whatever small instruct model is current when you read this.
model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# 50 toy preference pairs. A real run uses thousands of human-judged examples.
row = {
    "prompt":   "Suggest a quick vegetarian dinner.",
    "chosen":   "Paneer stir-fry over rice: saute paneer and veggies, add soy sauce. ~15 min.",
    "rejected": "You could make something using vegetables, prepared however you like.",
}
dataset = Dataset.from_list([row] * 50)

config = DPOConfig(
    beta=0.1,                          # the KL leash strength, same beta as the toy demo
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=5e-5,
    logging_steps=5,
    output_dir="dpo-toy",
)

# TRL builds the frozen reference model automatically when you do not pass one.
trainer = DPOTrainer(model=model, args=config, train_dataset=dataset, processing_class=tok)
trainer.train()

▶ Example output (toy run on a small GPU)

{'loss': 0.6931, 'rewards/chosen': 0.0000, 'rewards/rejected': 0.0000, 'rewards/margins': 0.0000, 'epoch': 0.2}
{'loss': 0.6802, 'rewards/chosen': 0.0141, 'rewards/rejected': -0.0138, 'rewards/margins': 0.0279, 'epoch': 0.4}
{'loss': 0.6631, 'rewards/chosen': 0.0325, 'rewards/rejected': -0.0361, 'rewards/margins': 0.0686, 'epoch': 0.6}
{'loss': 0.6448, 'rewards/chosen': 0.0510, 'rewards/rejected': -0.0602, 'rewards/margins': 0.1112, 'epoch': 0.8}
{'loss': 0.6285, 'rewards/chosen': 0.0693, 'rewards/rejected': -0.0847, 'rewards/margins': 0.1540, 'epoch': 1.0}
{'train_runtime': 41.7, 'train_samples_per_second': 1.2, 'epoch': 1.0}

What happened here: The TRL log tells the exact same story your toy loop did, which is the point of showing both. The loss starts near 0.6931 (log 2, the no-opinion baseline), rewards/chosen rises, rewards/rejected falls, and rewards/margins (chosen minus rejected) grows step by step. Those three reward numbers are the health check you watch on any real DPO run: if margins are not climbing, your data or your beta is wrong. Everything you learned from the fifteen-line PyTorch demo transfers directly to the production trainer, because it is running the same loss on real log-probabilities instead of hand-picked ones.

GRPO and the Reasoning Wave

RLHF and DPO were built mainly to align tone, helpfulness, and safety. A newer wave aims preference-style training at reasoning: getting a model to think through math, code, and logic step by step. The method getting the most attention at the time of writing is GRPO (Group Relative Policy Optimization), which came out of the DeepSeek reasoning models. Instead of a separate reward model scoring single answers, GRPO samples a group of answers to the same prompt, scores them (often against a checkable answer, like whether the code passes or the math is correct), and pushes the model toward the ones that scored better relative to their group.

It keeps the RL spirit of RLHF but drops the separately trained reward model, using the group’s own spread as the baseline.

You will also hear about variants and cousins with names that come and go: rejection sampling fine-tuning, online DPO, and various “RL from verifiable rewards” setups. Do not memorize the alphabet soup. The durable idea is that preference and reward signals, whether from a human, a learned model, or an automatic checker, are used to move a policy toward better outputs while a leash keeps it grounded. Whatever new acronym is trending when you read this, ask the same three questions: where does the reward come from, is there a separate reward model or not, and what keeps the model from drifting. Those answers place any new method on the map you already have.

Common Mistakes

Best Practices

Conclusion

So that is RLHF vs DPO, from the data up. Both start from the same preference pairs and aim at the same goal, aligning a model with what people actually prefer. RLHF trains a reward model and runs a reinforcement learning loop with a KL leash, which is powerful but has many moving parts and a real risk of reward hacking. DPO folds that entire objective into one supervised-style loss, keeps the leash through a frozen reference model, and drops the reward model and the RL loop, which is why it became the popular default.

You saw the DPO loss run for real and watched the chosen reward rise while the rejected reward fell, which is the single most important behaviour to recognize on any preference-tuning run.

The practical takeaway is that the concepts outlast the tools, and that includes the RLHF vs DPO debate itself. Reward signals, a policy, and a leash back to a reference will still be the vocabulary long after today’s trainer APIs and trendy algorithms like GRPO have rotated out. Learn the shape and every new method becomes a variation you can place, not a mystery. To see where this stage sits in the full picture, revisit how LLMs are trained, and for the whole path from Python basics to production AI, head to the Python + AI/ML Cookbook tutorial series home.

Frequently Asked Questions

Interview Questions on RLHF vs DPO

How interviewers actually probe this topic: real scenarios, with answers you can say out loud.

Q: Explain the difference between RLHF and DPO to someone who knows SFT.

Both take preference pairs, chosen versus rejected answers, and align the model with them, but they differ in machinery. RLHF first trains a reward model to score answers, then runs reinforcement learning so the policy chases higher scores, with a KL penalty to a frozen reference so it does not drift or cheat. DPO proves that objective can be rewritten as a single loss trained with ordinary gradient descent, so it skips the reward model and the RL loop entirely while keeping the reference model for the same leash. DPO is simpler and cheaper; RLHF is more flexible but has more moving parts.

Q: Why does RLHF need a KL penalty against a reference model?

Because the reward model is imperfect and gameable. If you let the policy maximize reward freely, it finds cheap ways to fool the reward model, padded or sycophantic answers that score well without being better, which is reward hacking. The KL penalty measures how far the policy has moved from the pre-alignment reference model and penalizes large moves, so the policy can only improve in ways that stay close to sensible language. It is the leash that keeps optimization honest.

Q: In DPO there is no reward model, so where does the reward come from?

It is implicit. DPO shows that beta times the difference between the policy’s log-probability and the reference model’s log-probability behaves like a reward. The loss pushes that implicit reward up for chosen answers and down for rejected ones. So the reward model did not vanish, it got absorbed into the loss function through the policy-versus-reference log-probability ratio. That is the core trick that lets DPO drop the separate model.

Q: What does beta control in DPO, and what happens at the extremes?

Beta sets the strength of the KL leash, how far the policy is allowed to move from the reference. A very small beta lets the model move a lot, which can overfit the preference data or drift into low-quality text. A very large beta keeps the model glued to the reference, so it barely changes and the preferences have little effect. You tune beta by watching the reward margins in the logs: they should climb steadily, neither exploding nor staying flat.

Q: A model passed SFT but still gives rude or rambling answers. Walk through your fix.

This is a preference problem, not a knowledge or format problem, so preference tuning is the right tool. I would collect preference pairs where the chosen answer is polite and concise and the rejected one is rude or rambling, then run DPO on top of the SFT model with a frozen reference. I would watch that chosen reward rises and rejected reward falls and the margins grow. I would reach for DPO before RLHF because it is simpler and there is no reward model to train or reward hacking to police, moving to RLHF only if I needed finer control and had the infrastructure.

Q: Where do newer methods like GRPO fit, and how do you reason about them in an interview?

GRPO and similar methods extend the same idea toward reasoning tasks, where the reward can come from an automatic checker, such as whether code passes tests or a math answer is correct, rather than a human-trained reward model. GRPO samples a group of answers per prompt and pushes the model toward the ones that scored better relative to the group, keeping the RL spirit while dropping the separate reward model. In an interview I would not memorize the acronyms; I would place any new method by asking where its reward comes from, whether it uses a separate reward model, and what keeps the policy from drifting. Those three answers locate it on the same map as RLHF and DPO.

More in this series:

• GenAI: Fine-Tuning LLMs with LoRA, QLoRA, and PEFT
• Model Context Protocol (MCP): Connect Your AI to Anything
• Build an MCP Server in Python (FastMCP, Step by Step)

Background: How LLMs Are Trained: Pretraining, SFT, and RLHF

Next up: GenAI: How to Evaluate LLMs and Preference Tuning

Series Home: Python + AI/ML Cookbook. Complete Tutorial Series

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: GenAI: QLoRA Fine-Tuning with Unsloth on a Free Colab GPU

Next: GenAI: Building AI Agents with LangGraph, CrewAI, and Beyond

Series Home: Python + AI/ML Tutorial Series