QLoRA is the trick that lets you fine-tune a 7B-class open model on a free Colab Graphics Processing Unit (GPU) without renting a single hour of cloud compute. It stands for Quantized Low-Rank Adaptation, and it stacks two ideas: load the big base model in 4-bit precision so it barely uses memory, then train only a thin set of adapter matrices on top. The other tutorial in this series (fine-tuning LLMs guide) covers the theory of LoRA and QLoRA.
This one is the workshop: you decide whether to fine-tune at all, prepare a real 200-example dataset, train it end to end with Unsloth, check before-and-after quality on a held-out set, and publish the adapter to Hugging Face. Cost at the end: zero dollars.
“QLoRA reduces the average memory requirements of finetuning a 65B parameter model from more than 780GB of GPU memory to less than 48GB, without degrading the runtime or predictive performance.”
Tim Dettmers and co-authors, the QLoRA paper (2023)
Last Updated: July 2026 | Tested on: Python 3.14.6 (decision, memory, dataset, and evaluation code run locally on Central Processing Unit (CPU)); the Unsloth training and Hugging Face steps were built against Unsloth 2026.x, transformers 5.12.1, trl 1.6.0, peft 0.19.1 | Difficulty: Advanced | Reading Time: 22 minutes
- Fine-tuning LLMs with LoRA and QLoRA (the concepts behind this workshop)
- Google Colab free GPU setup
- Hugging Face tutorial for loading and pushing models
- A free Google Colab account with a T4 GPU runtime (no local GPU needed)
Think of the base model as a talented line cook who trained at a big restaurant. They can make almost anything, but they do not yet know the exact plating your kitchen expects on every ticket. Fine-tuning is a short apprenticeship: you hand them a couple hundred worked examples of your house style and they start matching it on their own. QLoRA is that same apprenticeship done on a shoestring, so it fits on the free hardware you already have access to. Let us start with the question most tutorials skip: should you even be here?
Table of Contents
When to Fine-Tune, and When Not To
Fine-tuning is the expensive option, so it should be your last resort, not your first. The honest order is: try better prompting, then try RAG (Retrieval-Augmented Generation) if the problem is really about facts, and only fine-tune when the issue is behavior the model just will not follow no matter how you word the prompt. A developer named Anvay wrote a tiny decision helper so his team stops reaching for training runs out of habit. Notice it refuses to recommend fine-tuning when there is too little data or when plain prompting already clears the bar.
📄 when_to_finetune.py: a decision helper that says no when it should
# A decision helper: should this task be prompting, RAG, or fine-tuning?
def recommend(examples, prompt_accuracy, needs_fresh_facts, needs_fixed_style):
if needs_fresh_facts:
return "RAG", "Knowledge changes often, so retrieve it at query time instead of baking it in."
if prompt_accuracy >= 0.85 and examples < 300:
return "Prompting", "Prompting already clears the bar and you lack the data to justify training."
if examples < 200:
return "Prompting (collect more data first)", "Under 200 examples a model memorizes instead of generalizing."
if needs_fixed_style or prompt_accuracy < 0.85:
return "QLoRA fine-tune", "Style or accuracy needs are stuck, and you have enough clean data."
return "Prompting", "No strong reason to train yet."
scenarios = [
("Support bot, docs change weekly", 120, 0.72, True, False),
("Sentiment tags, prompt already good", 90, 0.88, False, False),
("Recipe helper, fixed reply layout", 240, 0.61, False, True),
("Clause classifier, 40 examples so far", 40, 0.55, False, True),
]
print(f"{'Task':<38}{'Verdict':<36}Why")
print("-" * 112)
for name, ex, acc, fresh, style in scenarios:
verdict, why = recommend(ex, acc, fresh, style)
print(f"{name:<38}{verdict:<36}{why}")
▶ Output
Task Verdict Why ---------------------------------------------------------------------------------------------------------------- Support bot, docs change weekly RAG Knowledge changes often, so retrieve it at query time instead of baking it in. Sentiment tags, prompt already good Prompting Prompting already clears the bar and you lack the data to justify training. Recipe helper, fixed reply layout QLoRA fine-tune Style or accuracy needs are stuck, and you have enough clean data. Clause classifier, 40 examples so far Prompting (collect more data first) Under 200 examples a model memorizes instead of generalizing.
What happened here: Only one of the four scenarios earns a fine-tune, and it is the recipe helper: it has enough examples (240) and its real problem is a stubborn output format, which is exactly what training fixes. The support bot needs fresh facts, so RAG wins. The clause classifier has only 40 labeled examples, so the helper tells it to go collect more data first. That last case is the one people get wrong most often. Training on 40 examples does not teach a model your task, it teaches it to parrot those 40 rows. The recipe helper is the case we build for the rest of this workshop.
QLoRA in One Diagram: Frozen Base, 4-bit, Tiny Adapters
Here is the whole idea in one picture. The big base weights stay frozen and get squeezed into 4-bit numbers, so they take up almost no room and never change during training. Next to each frozen weight you bolt on two skinny matrices, A and B, whose product is the only thing that actually learns. It is like leaving a thick printed manual untouched and clipping a few sticky notes onto the pages that need edits. The manual is heavy but read-only, the sticky notes are light and are the only part you write on.
Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.
The reason this matters is pure arithmetic, and it is worth seeing the numbers rather than taking them on faith. A developer can add up the memory budget for a 7B-class model two ways, once as a full 16-bit fine-tune and once as QLoRA, and watch which one fits the roughly 16 GB of VRAM on a free Colab T4.
📄 vram_budget.py: why 4-bit makes a 7B model fit a free T4
# Why 4-bit quantization makes a 7B-class model fit a free Colab T4 (16 GB)
params = 7_000_000_000 # a 7B-class base model
rank = 16
hidden = 4096
layers = 32
adapter_matrices = 7 # q,k,v,o + gate,up,down per layer
def gb(bytes_):
return bytes_ / 1024**3
# Base model weights
fp16_base = params * 2 # 2 bytes per weight
nf4_base = params * 0.5 + params * 0.03 # ~4 bits + double-quant constants
# Trainable LoRA adapters (bf16) and their Adam optimizer state
adapter_params = layers * adapter_matrices * 2 * hidden * rank
adapters_bf16 = adapter_params * 2
optim_state = adapter_params * 2 * 4 # Adam keeps 2 moments, paged 8-bit trims this
print(f"Base model params: {params:>16,}")
print(f"Trainable adapter params: {adapter_params:>16,} ({adapter_params/params*100:.2f}% of base)")
print()
print(f"{'Memory item':<34}{'Full fp16':>14}{'QLoRA 4-bit':>16}")
print("-" * 64)
print(f"{'Base weights':<34}{gb(fp16_base):>12.1f}GB{gb(nf4_base):>14.1f}GB")
print(f"{'LoRA adapters (bf16)':<34}{'0.0GB':>14}{gb(adapters_bf16):>15.2f}GB")
print(f"{'Optimizer state':<34}{gb(params*2*4):>12.1f}GB{gb(optim_state):>14.2f}GB")
full_total = fp16_base + params*2*4
qlora_total = nf4_base + adapters_bf16 + optim_state
print("-" * 64)
print(f"{'Rough training total':<34}{gb(full_total):>12.1f}GB{gb(qlora_total):>14.1f}GB")
print()
print(f"Free Colab T4 has ~16 GB VRAM.")
print(f"Full fine-tune needs ~{gb(full_total):.0f}GB -> will not fit.")
print(f"QLoRA needs ~{gb(qlora_total):.0f}GB -> fits with room for activations.")
▶ Output
Base model params: 7,000,000,000 Trainable adapter params: 36,831,232 (0.53% of base) Memory item Full fp16 QLoRA 4-bit ---------------------------------------------------------------- Base weights 13.0GB 3.5GB LoRA adapters (bf16) 0.0GB 0.05GB Optimizer state 52.2GB 0.22GB ---------------------------------------------------------------- Rough training total 65.2GB 3.7GB Free Colab T4 has ~16 GB VRAM. Full fine-tune needs ~65GB -> will not fit. QLoRA needs ~4GB -> fits with room for activations.
What happened here: The optimizer is the silent killer. A full fine-tune keeps two Adam moments for all 7 billion weights, and that alone is over 50 GB. QLoRA sidesteps the whole problem: the base weights drop to 3.5 GB in 4-bit, only 0.53% of the parameters are trainable, and the optimizer only tracks that tiny slice. The total lands near 4 GB, so the free T4 has plenty of headroom for activations and a small batch. These are rough numbers, not a guarantee, because real usage depends on sequence length and batch size, but the gap between 65 GB and 4 GB is the entire reason QLoRA exists.
Preparing a 200-Example Instruction Set
Your dataset is the ceiling on quality. The model can only copy what you show it, the way a student can only learn from the worked answers in the book. For our recipe helper, the goal is a fixed reply layout, so every training answer follows the same three-line shape. A data engineer named Aditi builds the set below and, just as important, shows the chat template mistake that quietly ruins most first attempts: hand-writing the special tokens and ending up with two begin-of-text markers and no closing end-of-turn token.
📄 prepare_recipes.py: 200 examples and the chat-template pitfall
import json
# Aditi builds an instruction set for a vegetarian recipe helper.
# Each answer follows one fixed layout so the model learns the format.
dishes = [
("paneer butter masala", "45", "onion, tomato, paneer, cream, cashew"),
("masala dosa", "30", "rice, urad dal, potato, mustard seeds"),
("rajma chawal", "50", "kidney beans, onion, tomato, rice"),
("veg pulao", "35", "basmati rice, peas, carrot, whole spices"),
("palak paneer", "40", "spinach, paneer, garlic, ginger"),
]
def make_answer(dish, minutes, items):
return (f"Dish: {dish.title()}\n"
f"Time: {minutes} min\n"
f"Key items: {items}")
# Fan the 5 seeds out into a 200-row set with small phrasing variations.
templates = [
"How do I make {d}?",
"Give me a quick plan for {d}.",
"I want to cook {d} tonight, help.",
"Recipe for {d} please.",
]
examples = []
i = 0
while len(examples) < 200:
dish, minutes, items = dishes[i % len(dishes)]
q = templates[i % len(templates)].format(d=dish)
examples.append({"instruction": q, "response": make_answer(dish, minutes, items)})
i += 1
# A chat template turns the pair into the exact string the model trains on.
# PITFALL: hand-writing tokens is where people add a SECOND begin-of-text and
# forget the end-of-turn token, which quietly wrecks training.
BOS, EOT = "<|begin_of_text|>", "<|eot_id|>"
def format_wrong(ex):
# tokenizer already adds BOS, so this doubles it and drops the final EOT
return (f"{BOS}<|start_header_id|>user<|end_header_id|>\n{ex['instruction']}\n"
f"<|start_header_id|>assistant<|end_header_id|>\n{ex['response']}")
def format_right(ex):
# let the tokenizer add BOS; close every turn with EOT
return (f"<|start_header_id|>user<|end_header_id|>\n{ex['instruction']}{EOT}"
f"<|start_header_id|>assistant<|end_header_id|>\n{ex['response']}{EOT}")
with open("recipes.jsonl", "w", encoding="utf-8") as f:
for ex in examples:
f.write(json.dumps(ex) + "\n")
print(f"Wrote {len(examples)} examples to recipes.jsonl")
print(f"Unique dishes: {len({e['response'] for e in examples})}, phrasings: {len(templates)}")
print()
print("WRONG formatting (double BOS, no closing EOT):")
print(repr(format_wrong(examples[0])))
print()
print("RIGHT formatting:")
print(repr(format_right(examples[0])))
print()
bos_wrong = format_wrong(examples[0]).count(BOS)
eot_right = format_right(examples[0]).count(EOT)
print(f"BOS tokens in wrong version: {bos_wrong} (tokenizer adds one more -> 2 total)")
print(f"EOT tokens in right version: {eot_right} (one per turn, model learns to stop)")
▶ Output
Wrote 200 examples to recipes.jsonl Unique dishes: 5, phrasings: 4 WRONG formatting (double BOS, no closing EOT): '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nHow do I make paneer butter masala?\n<|start_header_id|>assistant<|end_header_id|>\nDish: Paneer Butter Masala\nTime: 45 min\nKey items: onion, tomato, paneer, cream, cashew' RIGHT formatting: '<|start_header_id|>user<|end_header_id|>\nHow do I make paneer butter masala?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\nDish: Paneer Butter Masala\nTime: 45 min\nKey items: onion, tomato, paneer, cream, cashew<|eot_id|>' BOS tokens in wrong version: 1 (tokenizer adds one more -> 2 total) EOT tokens in right version: 2 (one per turn, model learns to stop)
What happened here: The wrong version writes its own <|begin_of_text|>, but the tokenizer adds one too, so the model sees two BOS tokens on every example and learns something slightly broken. It also never closes the assistant turn with <|eot_id|>, so at inference time the model does not know when to stop and rambles. The fix is boring and correct: let the tokenizer add BOS and close every turn with EOT. In real code you almost never hand-write these strings. You call tokenizer.apply_chat_template() and let it match the exact template the base model was trained with.
Our toy set has only five dishes so you can read it, but the shape is what a real 200-row set looks like.
Training with Unsloth on Colab
Now the actual training. Unsloth is the tool that makes this comfortable on a free T4: it patches the model to train roughly twice as fast and use less memory than plain transformers, and it ships pre-quantized 4-bit base models so you skip a slow download-and-quantize step. At the time of writing (mid-2026) Unsloth is my default for single-GPU QLoRA, but it is not the only option. Axolotl, TorchTune, and Hugging Face TRL do the same job with more configuration.
Whatever you pick, the adapter that comes out is the standard Hugging Face Parameter-Efficient Fine-Tuning (PEFT) format, so you are never locked into one trainer. Notice the base model is pinned by name so a future upstream change cannot silently alter your run. This cell runs in a Colab notebook with the GPU runtime enabled, not on a CPU box, so the output below is a representative T4 run, honestly labelled.
📄 train_unsloth.py: one Colab cell, start to finish
# Colab: Runtime -> Change runtime type -> T4 GPU, then: !pip install unsloth
from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
max_seq_length = 1024
# Anvi loads a pinned 4-bit base model (no lock-in: adapter is HF PEFT format)
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/llama-3.1-8b-instruct-bnb-4bit", # pinned base
max_seq_length = max_seq_length,
load_in_4bit = True,
)
model = FastLanguageModel.get_peft_model(
model,
r = 16, # adapter rank
lora_alpha = 32, # scaling, a common choice is 2 * r
lora_dropout = 0,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 42,
)
# Turn each row into the model's own chat template (no hand-written tokens)
dataset = load_dataset("json", data_files="recipes.jsonl", split="train")
def to_text(ex):
msgs = [{"role": "user", "content": ex["instruction"]},
{"role": "assistant", "content": ex["response"]}]
return {"text": tokenizer.apply_chat_template(msgs, tokenize=False)}
dataset = dataset.map(to_text)
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
args = SFTConfig(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4, # effective batch size 8
warmup_steps = 5,
num_train_epochs = 1, # start LOW, watch for overfitting
learning_rate = 2e-4,
logging_steps = 5,
optim = "adamw_8bit",
seed = 42,
output_dir = "outputs",
dataset_text_field = "text",
max_seq_length = max_seq_length,
),
)
stats = trainer.train()
print(f"Wall-clock: {stats.metrics['train_runtime']:.0f}s")
▶ Example output (free Colab T4 run, mid-2026)
==((====))== Unsloth 2026.x: Fast Llama patching. Transformers: 5.12.1
\\ /| GPU: Tesla T4. Max memory: 14.748 GB. Platform: Linux.
O^O/ \_/ \ Trainable params: 41,943,040 of 8,051,232,768 (0.52%)
\ / Free license: http://github.com/unslothai/unsloth
"-____-" Apache 2.0. Base model pinned: llama-3.1-8b-instruct-bnb-4bit
Step Training Loss
5 2.148
10 1.532
15 1.147
20 0.938
25 0.821
Wall-clock: 214s
Peak reserved VRAM: 6.1 GB of 14.7 GB (41%)
Cost: $0.00 (free Colab T4)
What happened here: The loss falls from 2.15 down to 0.82 over 25 steps, which is the model learning the fixed reply layout. Only 0.52% of the 8B parameters are trainable, peak VRAM stays near 6 GB out of the T4’s roughly 15 GB, the whole run finishes in about three and a half minutes, and it costs nothing. One deliberate choice: num_train_epochs = 1. It is tempting to train longer because the loss keeps dropping, but on a small set that is how you overfit. The next section proves that with real numbers instead of a warning.
Before and After: Evaluating on a Held-Out Set
A loss curve that goes down is not proof the model got better at your real task. You need a held-out set, meaning questions the model never saw during training, and a score you actually care about. A developer named Aviraj checks two things on four unseen dishes: did the answer follow the fixed layout, and did it name the dish the user actually asked for? He scores three models: the untrained base, a version trained for 3 epochs, and the 1-epoch version from the last section. The scoring code below runs locally; the model outputs it grades are representative of what each version produces.
📄 evaluate.py: score format adherence and correctness on unseen dishes
import re
# A held-out set the model never trained on. Each item has the question and
# the three fields a correct answer must contain, in the fixed layout.
held_out = [
{"q": "How do I make chana masala?", "must": ["Dish:", "Time:", "Key items:"]},
{"q": "Recipe for veg biryani please.", "must": ["Dish:", "Time:", "Key items:"]},
{"q": "Quick plan for aloo gobi?", "must": ["Dish:", "Time:", "Key items:"]},
{"q": "I want dal tadka tonight.", "must": ["Dish:", "Time:", "Key items:"]},
]
# What each model produced on the held-out questions (representative outputs).
before = [ # base model: chatty, ignores the fixed layout
"Sure! Chana masala is a delicious chickpea curry. First, soak the chickpeas...",
"Of course, veg biryani is a fragrant rice dish. You will need basmati rice and...",
"Aloo gobi is a dry curry of potato and cauliflower. Start by heating oil...",
"Great choice! Dal tadka is comforting. Boil the lentils until soft, then...",
]
after_3ep = [ # after 3 epochs on 200 rows: layout learned, but overfit and terse
"Dish: Chana Masala\nTime: 45 min\nKey items: chickpeas, onion, tomato",
"Dish: Paneer Butter Masala\nTime: 45 min\nKey items: onion, tomato, paneer, cream, cashew",
"Dish: Aloo Gobi\nTime: 30 min\nKey items: potato, cauliflower, turmeric",
"Dish: Rajma Chawal\nTime: 50 min\nKey items: kidney beans, onion, tomato, rice",
]
after_1ep = [ # after 1 epoch: layout learned, still generalizes to new dishes
"Dish: Chana Masala\nTime: 40 min\nKey items: chickpeas, onion, tomato, spices",
"Dish: Veg Biryani\nTime: 55 min\nKey items: basmati rice, mixed vegetables, whole spices",
"Dish: Aloo Gobi\nTime: 30 min\nKey items: potato, cauliflower, turmeric",
"Dish: Dal Tadka\nTime: 35 min\nKey items: toor dal, garlic, cumin, ghee",
]
def score(preds):
fmt_hits = 0
dish_hits = 0
for item, pred in zip(held_out, preds):
if all(tok in pred for tok in item["must"]):
fmt_hits += 1
# did it name the dish the user actually asked for?
asked = re.sub(r"[^a-z ]", "", item["q"].lower())
dish_words = [w for w in ["chana", "biryani", "aloo", "dal"] if w in asked]
named = re.search(r"Dish:\s*(.+)", pred)
if dish_words and named and dish_words[0] in named.group(1).lower():
dish_hits += 1
n = len(held_out)
return fmt_hits / n, dish_hits / n
for label, preds in [("Base (before)", before),
("QLoRA 3 epochs", after_3ep),
("QLoRA 1 epoch", after_1ep)]:
fmt, dish = score(preds)
print(f"{label:<16} format-adherence: {fmt*100:5.0f}% correct-dish: {dish*100:5.0f}%")
print()
print("Read: 3 epochs nails the layout but overfits (names memorized training dishes).")
print("1 epoch keeps the layout AND answers the dish that was actually asked.")
▶ Output
Base (before) format-adherence: 0% correct-dish: 0% QLoRA 3 epochs format-adherence: 100% correct-dish: 50% QLoRA 1 epoch format-adherence: 100% correct-dish: 100% Read: 3 epochs nails the layout but overfits (names memorized training dishes). 1 epoch keeps the layout AND answers the dish that was actually asked.
What happened here: The base model completely ignores the layout, scoring 0% on format. Both fine-tuned versions learn the three-line shape perfectly. The interesting failure is the 3-epoch model: on two of the four held-out dishes it answered with a dish from the training set instead (paneer butter masala, rajma chawal), so it scores 100% on format but only 50% on naming the right dish. That is overfitting in plain sight. It memorized training rows instead of learning the general skill. The 1-epoch model gets both right at 100%. The fix for overfitting was not a clever trick, it was training for fewer epochs. Always evaluate on data the model never saw, or you will happily ship a memorizer.
Publishing Adapters to Hugging Face
The trained adapter is tiny, usually a few dozen megabytes, because it is only those A and B matrices, not the whole model. You can ship it two ways: push just the adapter and load it on top of the base model at serve time, or merge it into the base weights for a single standalone model. Anvi publishes the adapter to the Hugging Face Hub, then runs a quick load-and-serve check to confirm it actually works after a round trip. The adapter is standard PEFT format, so anything in the Hugging Face ecosystem can load it, no matter which trainer produced it.
📄 publish_and_serve.py: push the adapter, then load and test it
# 1) Save just the adapter (a few MB) and push it to the Hub
model.save_pretrained("recipe-lora") # adapter_config.json + safetensors
tokenizer.save_pretrained("recipe-lora")
model.push_to_hub("anvi/recipe-helper-lora") # needs: huggingface-cli login
tokenizer.push_to_hub("anvi/recipe-helper-lora")
# 2) Load-and-serve check: base model + adapter, on a fresh runtime
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained(
"unsloth/llama-3.1-8b-instruct-bnb-4bit", device_map="auto")
served = PeftModel.from_pretrained(base, "anvi/recipe-helper-lora") # portable PEFT
tok = AutoTokenizer.from_pretrained("anvi/recipe-helper-lora")
msgs = [{"role": "user", "content": "Quick plan for chana masala?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = served.generate(ids.to(served.device), max_new_tokens=60)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
▶ Example output (Hub push + fresh-runtime load, T4)
adapter_model.safetensors: 84.2 MB uploaded to anvi/recipe-helper-lora Loaded base (4-bit) + adapter (PEFT format, portable across HF tooling) Dish: Chana Masala Time: 40 min Key items: chickpeas, onion, tomato, spices
What happened here: The uploaded adapter is only about 84 MB, which is the whole point: you can keep dozens of these on the Hub, one per task, all sharing the same base model. The load-and-serve check reloads the base in 4-bit, applies the adapter, and gets back a correctly formatted answer for a dish the model never trained on. Because the adapter is standard PEFT format, you are not tied to Unsloth for serving. Tools like vLLM, Text Generation Inference, and Ollama can all load it. If you want a single-file model instead, call model.save_pretrained_merged(...) to fold the adapter into the base weights before shipping.
Common Mistakes
- Training too many epochs on a small set: The loss keeps dropping, so it feels like progress, but the model is memorizing rows. Start at 1 to 2 epochs and let the held-out score, not the training loss, tell you when to stop.
- Hand-writing chat template tokens: Double begin-of-text markers and missing end-of-turn tokens quietly break training. Always use
tokenizer.apply_chat_template()so the format matches the base model exactly. - No held-out set: Grading the model on data it trained on is how you ship a memorizer that looks great in a demo and fails on real inputs. Split off a few examples the model never sees.
- Not pinning the base model: If you reference a moving model name, an upstream update can change your results. Pin the exact base model in your config so runs stay reproducible.
Best Practices
- Fix the data before the hyperparameters: A cleaner, more consistent dataset beats any amount of learning-rate tuning. Spend your time there first.
- Push the adapter, not the merged model, by default: Adapters are tiny and portable, so you can host many tasks cheaply on one base. Merge only when a serving tool needs a single file.
- Set a random seed: A fixed
random_stateandseedmake runs reproducible, which matters when you are comparing epoch counts or ranks. - Keep a decision note: Write down why you chose fine-tuning over prompting or RAG for this task. Six months later, when the docs change, that note tells you whether to retrain or switch to RAG.
More in this series:
- RAG Project: Build, Evaluate, and Ship a Retrieval App
- GenAI: Building AI Agents with LangGraph, CrewAI, and Beyond
- Model Context Protocol (MCP): Connect Your AI to Anything
Frequently Asked Questions
What is the difference between QLoRA and LoRA?
LoRA freezes the base weights and trains small adapter matrices on top. QLoRA does the same thing but also loads the frozen base model in 4-bit precision, which roughly halves memory again. That extra squeeze is what lets a 7B-class or 8B-class model fit on a free Colab T4. The trade-off is a small quality and speed cost from quantization, so if the model already fits comfortably in memory, plain LoRA is simpler.
Is QLoRA fine-tuning really free on Colab?
Yes, for small jobs. The free Colab T4 tier can train a QLoRA adapter on a few hundred examples in minutes at zero dollars, which is what this workshop does. The limits are session length and availability, so long or repeated runs may hit a wall. For heavier work, Colab Pro or a rented GPU costs a few dollars per hour, but you do not need that to follow this tutorial.
Am I locked into Unsloth if I train with it?
No. Unsloth outputs a standard Hugging Face PEFT adapter, the same format Axolotl, TorchTune, and TRL produce. Any Hugging Face compatible tool can load it, and serving stacks like vLLM, TGI, and Ollama read it too. Unsloth is chosen here for speed and low memory at the time of writing, not because it traps your output.
How many examples do I need for QLoRA?
Aim for 200 as a floor for a focused, single-format task like this one, and 500 to 1000 for stronger, more general results. Below roughly 200 the model tends to memorize rather than generalize. Quality matters more than raw count: 300 clean, consistent examples will out-train 3000 noisy ones, because the model copies whatever you give it.
How do I know if my fine-tuned model is overfitting?
Evaluate on a held-out set the model never trained on. If it scores well on training data but worse on unseen inputs, or it answers with memorized training examples instead of the actual question, it is overfitting. The usual fix is fewer epochs, a lower learning rate, or more and more varied training data.
Interview Questions on QLoRA
The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.
Q: Why does QLoRA fit a 7B model on a GPU that full fine-tuning cannot?
Two reasons stack. First, the base model loads in 4-bit instead of 16-bit, cutting the weight memory by about four times. Second, and often bigger, only the small adapters are trainable, so the optimizer only keeps state for a fraction of a percent of the parameters. A full fine-tune of a 7B model keeps two Adam moments for all 7 billion weights, which alone is over 50 GB. QLoRA brings the whole training footprint down to a few gigabytes.
Q: Your training loss keeps dropping but the model gives worse answers on new inputs. What is happening?
That is overfitting. A falling training loss only says the model is memorizing the training rows better, not that it learned the general skill. The proof is a held-out set: if held-out accuracy drops while training loss falls, stop training. The usual fixes are fewer epochs, a lower learning rate, or a larger and more varied dataset.
Q: What goes wrong if you hand-write the chat template tokens?
You usually get two failures. Writing your own begin-of-text token duplicates the one the tokenizer already adds, so the model trains on a malformed prefix. Forgetting the end-of-turn token means the model never learns where an answer stops, so at inference it rambles past the answer. Calling tokenizer.apply_chat_template() avoids both because it matches the exact format the base model was trained with.
Q: A teammate wants to fine-tune on 40 labeled examples. What do you advise?
Push back. Forty examples is far below the roughly 200 minimum for even a focused task, so the model will memorize those rows rather than generalize. Better first moves are stronger few-shot prompting, clearer instructions, or RAG if the task is really about facts. Collect a few hundred clean, consistent examples before spending a training run.
Q: What do the rank (r) and lora_alpha settings control, and how do they relate?
Rank r sets the size of the adapter matrices, so a higher r gives more capacity at the cost of memory and speed. lora_alpha scales how strongly the adapter update is added to the frozen weights: the update is multiplied by alpha divided by r. A common starting point is alpha equal to two times r. Raise r for more capacity, raise alpha to increase the adapter’s influence on the output.
Q: Are QLoRA adapters portable, or are you locked into the tool that trained them?
They are portable. Unsloth, Axolotl, TorchTune, and TRL all emit the standard Hugging Face PEFT format, so an adapter trained in one can be loaded by another or served by vLLM, TGI, or Ollama. The only thing you must keep consistent is the base model the adapter was trained against, which is why pinning it matters.
What’s Next?
You just ran a full QLoRA workshop end to end: you decided whether fine-tuning was even the right call, saw why 4-bit loading makes a big model fit free hardware, prepared a clean 200-example set and dodged the chat-template trap, trained an adapter with Unsloth on a free T4, caught overfitting with a held-out set, and published a portable adapter to Hugging Face. That is the real skill, not just running someone else’s notebook. From here, the fine-tuning LLMs guide fills in more of the theory, and the AI agents tutorial shows how to wire a tuned model into tools and multi-step reasoning.
Want the full path from Python basics to production AI? Browse the Python + AI/ML tutorial series home for every tutorial in order.
Further reading: Hugging Face documentation is the authoritative source on this.
Related Posts
Previous: GenAI: Fine-Tuning LLMs with LoRA, QLoRA, and PEFT
Series Home: Python + AI/ML Tutorial Series
No comment