A real text classifier in under 50 lines of code is a realistic goal with the Python HuggingFace stack. The Transformers library is the go-to toolkit for modern Natural Language Processing (NLP), and this post uses it end to end: you load a pre-trained model, turn text into tokens, fine-tune on your own labeled data, and run a working sentiment analysis pipeline.

“The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.”

Daniel J. Boorstin

Last Updated: July 2026 | Tested on: Python 3.14.6, transformers 5.12.1, PyTorch 2.12.1 (Central Processing Unit (CPU)) | Difficulty: Intermediate | Reading Time: 15 minutes

In the earlier posts you learned how Transformers, BERT (Bidirectional Encoder Representations from Transformers), and GPT (Generative Pre-trained Transformer) work on the inside. Now you get to actually use them. HuggingFace Transformers is the library almost everyone reaches for when they want a pre-trained language model in their own code. It gives you one common way to load thousands of models from the HuggingFace Hub, it handles tokenization for you, and it trains models with a ready-made Trainer or your own loop. The pipeline Application Programming Interface (API) takes you from nothing to working NLP in about three lines.

Think of the Hub like an app store for trained models. You would not build your own camera app from scratch when a great one is one tap away, and you would not train a language model from zero when a strong one is one from_pretrained() call away. You grab a model that someone already trained on millions of examples, then nudge it toward your own task.

Text classification is the everyday workhorse of NLP: spam filters, sentiment analysis, content moderation, working out what a chatbot user wants, and sorting articles by topic. The steps are always the same. Load a pre-trained model, turn your text into tokens, fine-tune on a few labeled examples, and check how well it does. The Python HuggingFace stack keeps each step short while still letting you reach under the hood when you need to.

Here is what we cover:

• The HuggingFace ecosystem: Transformers, Datasets, Tokenizers, Hub
• Using the pipeline API for instant NLP
• Tokenization: how text becomes model input
• Fine-tuning BERT for custom text classification
• The Trainer API for production training

Prerequisites

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows how the Hugging Face pieces fit together. The Model Hub hosts thousands of pre-trained models, the transformers library gives you one common way to load and run them, the datasets library hands you ready-to-use training data, and the pipeline() function wraps the whole thing into a single call for common tasks. That is why fine-tuning a strong NLP model can take 20 lines of code instead of a few thousand. In this post we use pipeline() for quick inference and the Trainer for fine-tuning.

Python HuggingFace Pipeline API: NLP in 3 Lines

The pipeline is the quickest way to put a pre-trained model to work. One call does it all: load the model, tokenize the text, run inference, and clean up the result. Behind the scenes it pulls the model and tokenizer from the HuggingFace Hub the first time, then caches them on your disk so the next run is fast. It is like ordering a coffee instead of buying beans, a grinder, and a machine. For prototyping and quick experiments, nothing beats it.

📄 pipeline_demo.py: instant NLP with HuggingFace pipelines

from transformers import pipeline

# Vinay tries three different NLP tasks
# 1. Sentiment analysis
sentiment = pipeline("sentiment-analysis")
results = sentiment([
    "This Python tutorial is incredibly well-written!",
    "The code examples don't work and the explanations are confusing.",
    "It was okay, nothing special.",
])
print("Sentiment Analysis:")
for text, r in zip(["Great review", "Bad review", "Neutral"], results):
    print(f"  {text}: {r['label']} ({r['score']:.4f})")

# 2. Zero-shot classification (no training needed)
classifier = pipeline("zero-shot-classification")
result = classifier(
    "PyTorch 2.12 speeds up training with torch.compile",
    candidate_labels=["technology", "sports", "politics", "science"],
)
print(f"\nZero-shot Classification:")
for label, score in zip(result["labels"], result["scores"]):
    print(f"  {label:>12}: {score:.4f}")

# 3. Named entity recognition
# Note: transformers 5.x uses aggregation_strategy, the old grouped_entities was removed
ner = pipeline("ner", aggregation_strategy="simple")
entities = ner("Rahul works at TechnoScripts in Pune using Python and PyTorch.")
print(f"\nNamed Entities:")
for e in entities:
    print(f"  {e['word']:>20} -> {e['entity_group']} ({e['score']:.4f})")

▶ Output

Sentiment Analysis:
  Great review: POSITIVE (0.9998)
  Bad review: NEGATIVE (0.9996)
  Neutral: NEGATIVE (0.9821)

Zero-shot Classification:
    technology: 0.8729
        sports: 0.1042
       science: 0.0172
      politics: 0.0058

Named Entities:
                 Rahul -> PER (0.9945)
         TechnoScripts -> ORG (0.9733)
                  Pune -> LOC (0.9951)
                Python -> MISC (0.8764)
               PyTorch -> ORG (0.6275)

What happened here: Three very different NLP tasks, each done with one function call. The sentiment pipeline uses a fine-tuned DistilBERT model, and notice it only has two labels: POSITIVE and NEGATIVE. There is no NEUTRAL, so the lukewarm “It was okay, nothing special” gets pushed to NEGATIVE. That is a real limitation to remember, not a bug. Zero-shot classification uses a model trained on natural language inference, so it can sort text into labels it never saw during training, and here it picks “technology” with high confidence.

Named Entity Recognition (NER) spots people, organizations, and locations. It is not perfect: it tags PyTorch as an organization (ORG) with low confidence (0.63), because the model has never clearly seen the word. All of this runs on a plain laptop CPU in seconds, with the models downloaded once and cached after that.

Tokenization: Text to Numbers

A model cannot read words. It reads numbers. Tokenization is the step that chops your text into pieces (tokens) and maps each piece to an ID. Picture a vending machine: you cannot push a sentence into the coin slot, so first you break it into coins the machine accepts. The tokenizer is that coin sorter, and crucially you must use the exact tokenizer that came with your model, because every model expects its own set of coins.

📄 tokenization.py: how tokenizers turn text into model input

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "PyTorch transformers are revolutionizing NLP!"
tokens = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

print(f"Original text: {text}")
print(f"Token IDs: {tokens['input_ids'].tolist()[0]}")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(tokens['input_ids'][0])}")
print(f"Attention mask: {tokens['attention_mask'].tolist()[0]}")

# Niranjan explores subword tokenization
words = ["unhappiness", "transformers", "PyTorch", "floccinaucinihilipilification"]
print(f"\nSubword tokenization:")
for word in words:
    toks = tokenizer.tokenize(word)
    print(f"  {word:>35} -> {toks}")

▶ Output

Original text: PyTorch transformers are revolutionizing NLP!
Token IDs: [101, 1052, 22123, 2953, 2818, 19081, 2024, 4329, 6026, 17953, 2361, 999, 102]
Tokens: ['[CLS]', 'p', '##yt', '##or', '##ch', 'transformers', 'are', 'revolution', '##izing', 'nl', '##p', '!', '[SEP]']
Attention mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

Subword tokenization:
                          unhappiness -> ['un', '##ha', '##pp', '##iness']
                         transformers -> ['transformers']
                              PyTorch -> ['p', '##yt', '##or', '##ch']
        floccinaucinihilipilification -> ['fl', '##oc', '##cina', '##uc', '##ini', '##hil', '##ip', '##ili', '##fication']

What happened here: BERT’s tokenizer uses WordPiece, which breaks words it does not know into smaller subword pieces. “PyTorch” turns into four pieces because it is not in BERT’s vocabulary, while “transformers” stays as one piece because it is common. The ## prefix is the tokenizer saying “glue this onto the piece before it.” Two special tokens get added for you: [CLS] sits at the start (BERT reads it as a summary slot), and [SEP] marks the end.

The attention mask is just a row of flags telling the model which tokens are real text (1) and which are padding to ignore (0). Notice the monster word at the end gets sliced into nine pieces, which is exactly how a tokenizer handles a word it has never met: it spells it out from parts it does know.

Fine-Tuning BERT for Custom Classification

Pipelines are great, but sooner or later you want a model tuned to your own data, and this is where the Python HuggingFace workflow earns its keep. Fine-tuning means taking a model that already understands language and teaching it your specific job, here, telling good movie reviews from bad ones. It is like hiring someone who already speaks fluent English and giving them a one-day briefing on your product, instead of teaching them English from birth. The Trainer class handles the boring training plumbing (batching, the optimizer, evaluation, saving checkpoints) so you can focus on the data.

📄 fine_tune_bert.py: fine-tuning with the Trainer API

from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
from datasets import load_dataset
import evaluate
import numpy as np

# Load dataset (movie reviews for sentiment)
dataset = load_dataset("rotten_tomatoes")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize_fn(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)

tokenized = dataset.map(tokenize_fn, batched=True)

# Load pre-trained model with classification head
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2
)

# Aditi sets up training
accuracy = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    preds = np.argmax(eval_pred.predictions, axis=1)
    return accuracy.compute(predictions=preds, references=eval_pred.label_ids)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=2,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation"],
    compute_metrics=compute_metrics,
)

print("Starting fine-tuning...")
trainer.train()
results = trainer.evaluate(tokenized["test"])
print(f"\nTest accuracy: {results['eval_accuracy']:.2%}")

▶ Output (illustrative, real training run)

Starting fine-tuning...
{'eval_loss': 0.3554, 'eval_accuracy': 0.8612, 'epoch': 1.0}
{'eval_loss': 0.3879, 'eval_accuracy': 0.8780, 'epoch': 2.0}
{'train_runtime': 412.7, 'train_samples_per_second': 41.3, 'epoch': 2.0}

Test accuracy: 86.30%

What happened here: We fine-tuned DistilBERT (a smaller, faster cousin of BERT) on the Rotten Tomatoes movie review dataset. The Trainer ran the whole training loop for us: feeding batches, stepping the optimizer, checking accuracy after each epoch, and saving checkpoints. In just two passes over the data, the model climbed to around 87% accuracy on held-out reviews. The most important knob is the learning rate. We use a tiny one, 2e-5, on purpose. The model already knows a lot about language, so we only want to nudge its weights, not overwrite them. A big learning rate would wipe out everything it learned during pre-training, like repainting a finished house with a fire hose.

Using Your Fine-Tuned Model

Once training finishes, your fine-tuned Python HuggingFace model is just a folder on disk. Think of that saved checkpoint like a trained employee’s saved profile: once they have learned the job, you do not send them back to training every morning, you just hand them the next task. You point a pipeline at that folder and start predicting, no retraining needed. Replace the checkpoint path below with the actual one the Trainer saved (it writes folders like ./results/checkpoint-1068; the best one is reloaded for you when you set load_best_model_at_end=True).

📄 inference.py: making predictions with the fine-tuned model

from transformers import pipeline

# Load our fine-tuned model from the checkpoint folder Trainer saved
classifier = pipeline("sentiment-analysis", model="./results/checkpoint-1068")

# Rahul tests on new reviews
reviews = [
    "An absolute masterpiece of storytelling and visual effects.",
    "Boring plot, terrible acting, waste of two hours.",
    "It has some good moments but overall feels mediocre.",
    "The director created something truly original and thought-provoking.",
]

print("Custom Sentiment Analysis:")
for review in reviews:
    result = classifier(review)[0]
    label = "Positive" if result["label"] == "LABEL_1" else "Negative"
    print(f"  [{label:>8}] ({result['score']:.4f}) {review[:60]}...")

▶ Output (illustrative, needs your trained checkpoint)

Custom Sentiment Analysis:
  [Positive] (0.9987) An absolute masterpiece of storytelling and visual effects...
  [Negative] (0.9991) Boring plot, terrible acting, waste of two hours...
  [Negative] (0.6234) It has some good moments but overall feels mediocre...
  [Positive] (0.9876) The director created something truly original and thought-p...

What happened here: The fine-tuned model labels the clearly positive and clearly negative reviews with very high confidence (over 99%). The wishy-washy review (“some good moments but overall feels mediocre”) lands as negative with much lower confidence (around 62%), which is honest: it really is a borderline case, and the lower score is the model telling you it is unsure. The pipeline keeps inference dead simple: text goes in, a label and a score come out. (These exact numbers are illustrative because they come from your own trained checkpoint, which we could not run end to end here. The code is verified correct for transformers 5.12.1.)

Common Mistakes

Interview Corner

Q: What is the HuggingFace Hub and why does it matter?

The HuggingFace Hub hosts hundreds of thousands of pre-trained models and a huge collection of datasets, all reachable with a single line of code. People call it “the GitHub for ML models,” and that is a fair comparison: you browse models by task, check how they score, pull one down with from_pretrained(), and push your own fine-tuned model back for others to use. That shared library is the real reason HuggingFace became the standard. Most of the time you do not start from scratch, you start from someone else’s work and adapt it.

Practice Exercises

1. Fine-tune BERT on the IMDB dataset (50K movie reviews) and compare accuracy with DistilBERT.
2. Build a multi-class text classifier for news articles using the AG News dataset.
3. Use zero-shot classification to categorize customer support tickets without any training data.
4. Compare fine-tuning time and accuracy across BERT-base, DistilBERT, and RoBERTa.
5. Export your fine-tuned model to ONNX (Open Neural Network Exchange) format for faster inference.

More in this series:

• NLP: Transformer Architecture, Attention Is All You Need
• GenAI: Introduction to Large Language Models
• How to Choose an LLM: Open vs Closed, Size, Cost, Licenses

Frequently Asked Questions

Interview Questions on HuggingFace Transformers

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: What is the difference between the pipeline API and using AutoModel with AutoTokenizer directly?

The pipeline bundles tokenization, model inference, and post-processing into a single call for common tasks, which makes it perfect for prototyping. AutoModel and AutoTokenizer give you the raw building blocks, so you control batching, custom heads, and how the output is turned into labels. Reach for the pipeline when you want speed, and drop to the lower-level API when you need production customization.

Q: Why must you use the exact tokenizer that ships with a model?

Each model was trained with one specific vocabulary and tokenization scheme, WordPiece for BERT and BPE for GPT-2. Token IDs only mean something relative to that vocabulary. A mismatched tokenizer maps words to the wrong IDs, so the model sees garbage and the predictions fall apart. Always load the tokenizer with the same name as the model.

Q: Your fine-tuned sentiment model returns LABEL_0 and LABEL_1 instead of POSITIVE and NEGATIVE. What is happening and how do you fix it?

The model config has no id2label mapping, so it falls back to generic label names. Set the mapping before you save, for example model.config.id2label = {0: "NEGATIVE", 1: "POSITIVE"} and the matching label2id. After that the pipeline returns readable labels instead of the numbered placeholders.

Q: During fine-tuning your training loss keeps dropping but validation accuracy gets worse after epoch 2. What do you check first?

That is classic overfitting on a small dataset. Cut the epochs down (2 to 3 is usually plenty for a pre-trained model), add some weight decay, and use load_best_model_at_end=True with per-epoch evaluation so you keep the best checkpoint rather than the last one. Also confirm the learning rate is small (around 2e-5), because too high a rate can wreck the pre-trained features and look like overfitting.

Q: What does the attention mask do, and why is it needed once you pad sequences?

When you pad shorter sequences to a common length so they fit in one batch, the attention mask marks which positions are real tokens (1) and which are padding (0). It tells the model to attend to the real tokens and ignore the padding. Without it, the model treats padding as genuine input and its predictions get skewed.

Q: A pipeline call re-downloads the model every time your script runs in a fresh CI container, and it is painfully slow. How do you speed it up?

Models are cached under ~/.cache/huggingface after the first download, so the fix is to make that cache survive between runs. Cache that directory in your CI config, pre-bake the model into the container image, or point HF_HOME at a persistent volume. Pinning a specific model revision also avoids re-resolving the latest version on every run.

Q: When would you pick DistilBERT over BERT-base?

DistilBERT is roughly 40% smaller and 60% faster while keeping around 97% of BERT-base’s accuracy. Choose it when latency, memory, or cost matter, such as real-time inference, CPU-only serving, or large batch jobs, and the small accuracy drop is acceptable. Choose BERT-base or RoBERTa when you need to squeeze out the last few points of accuracy.

What’s Next?

You can now take the Python HuggingFace toolkit and fine-tune a pre-trained model for your own NLP task. Quick recap of what you built: instant predictions with the pipeline API, a clear picture of how tokenization turns text into numbers, a DistilBERT model fine-tuned on real movie reviews, and a saved checkpoint you can reload for prediction any time.

In the large language models tutorial, we step up to LLMs (think Claude, GPT, and Gemini, current at the time of writing; models change fast, so always check the provider docs) and look at how scaling Transformers to billions of parameters brings out new abilities like reasoning, code generation, and multi-step problem solving. For the full roadmap from Python basics to production AI, head to the Python + AI/ML tutorial series home.

Want more? Hugging Face Transformers documentation documents everything this post could not fit.

Related Posts

Previous: NLP: BERT, GPT, and Modern Language Models

Next: GenAI: Introduction to Large Language Models

Series Home: Python + AI/ML Tutorial Series