A model cannot read. Feed it ten thousand product reviews and it sees nothing but bytes until you turn each sentence into numbers. That translation step is what NLP basics are really about: tokenize the text, drop the noise words, normalize what remains, and score every term with TF-IDF. This post walks that whole path with NLTK and scikit-learn, ending in a working classifier.

“There should be one, and preferably only one, obvious way to do it.”

Tim Peters, The Zen of Python (PEP 20)

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0, NLTK 3.9.4 | Difficulty: Intermediate | Reading Time: 20 minutes

Think of it like cooking. Raw text is a basket of muddy vegetables straight from the field. You cannot drop them in the pot as they are. First you wash off the dirt (punctuation and casing), you peel away the bits nobody eats (stop words like “the” and “is”), and you chop everything to a standard size (stemming or lemmatization, so “running”, “runs”, and “ran” all become one ingredient). Only then do you measure portions (vectorization) and start cooking (train a model). Skip the prep and the meal tastes terrible. The quality of your text prep sets the ceiling for every model that comes after it.

The star ingredient at the end is TF-IDF, short for Term Frequency times Inverse Document Frequency. It scores each word by how often it shows up in one document, then divides that down by how common the word is across every document. “The” appears everywhere, so it gets a tiny score. “Gradient” appears rarely, so in a machine learning article it gets a big score. TF-IDF has been the default text vectorizer for decades, and paired with a simple classifier it still holds its own against far heavier deep learning models on small and medium datasets.

Prerequisites

Install and Verify

Two libraries do almost all the work in this post. NLTK (the Natural Language Toolkit) handles the language-aware steps: splitting sentences, listing stop words, stemming, and lemmatizing. scikit-learn handles the math: turning words into TF-IDF vectors and training the classifier. Install both with pip.

📄 Terminal: install the two libraries

pip install nltk scikit-learn

NLTK ships almost empty on purpose. The big data files (tokenizer models, the stop word list, the WordNet dictionary) are downloaded separately, so you grab only what you need. Run this once and the files land in your home folder for good.

📄 download_data.py: fetch the NLTK data packs (run once)

import nltk

# Sentence and word tokenizer models, the stop word list, and WordNet
nltk.download("punkt_tab")
nltk.download("stopwords")
nltk.download("wordnet")

Now check that both libraries import and report the versions this post was tested against.

📄 verify.py: confirm your setup matches

import nltk
import sklearn

print("NLTK:", nltk.__version__)
print("scikit-learn:", sklearn.__version__)

▶ Output

NLTK: 3.9.4
scikit-learn: 1.9.0

What happened here: If you see two version lines and no traceback, you are ready. NLTK 3.9.4 and scikit-learn 1.9.0 are what the code below was run on. If the import fails, you are almost certainly in the wrong virtual environment, so activate the one where you ran pip and try again.

The Quick Win: Tokenizing in Three Lines

Before any theory, let us feel the tool work. Tokenizing means cutting a blob of text into pieces: sentences and words. You could try to do it with text.split(), but that breaks on punctuation (“okay.” becomes one weird token) and ignores sentence boundaries. NLTK already knows the rules of English, so it does the cutting properly.

📄 quickwin.py: split a review into sentences and words

from nltk.tokenize import word_tokenize, sent_tokenize

review = "The phone is amazing. Battery life is great, but the camera is just okay."

# Split into sentences
sentences = sent_tokenize(review)
print(f"Sentences ({len(sentences)}):")
for s in sentences:
    print(f"  - {s}")

# Split into words (punctuation becomes its own token)
words = word_tokenize(review)
print(f"\nWord tokens ({len(words)}):")
print(words)

▶ Output

Sentences (2):
  - The phone is amazing.
  - Battery life is great, but the camera is just okay.

Word tokens (17):
['The', 'phone', 'is', 'amazing', '.', 'Battery', 'life', 'is', 'great', ',', 'but', 'the', 'camera', 'is', 'just', 'okay', '.']

What happened here: Two function calls, real results. NLTK saw two sentences, splitting on the period after “amazing” but correctly NOT splitting on the comma. The word tokenizer pulled out 17 tokens and, notice, it treated the period and comma as their own tokens instead of gluing them to “okay” or “great”. That is exactly what you want, because “okay.” and “okay” should count as the same word later on. You felt the tool work in under two minutes. Now let us see the full pipeline it fits into.

The NLP Text Processing Pipeline

Read the diagram top to bottom. Raw text enters, gets split into word tokens, loses its stop words, and gets normalized by stemming or lemmatization. Then comes the fork that matters most: vectorization. You can count words plainly (bag of words) or weight them by importance (TF-IDF). Either way you end up with a feature matrix, a grid of numbers, that any machine learning model can train on. Every step before the fork is about throwing away noise so the model sees only signal. This five-step pipeline is the backbone of NLP basics; everything later in the series just swaps out the final box.

Here is a compact, stdlib-only version of the early steps so you can see the shape of the data change at each stage. No libraries beyond re and collections, which means you can run this on any plain Python install.

📄 text_preprocessing.py: from raw text to clean tokens

import re
from collections import Counter

# Raw text
text = """
Machine learning is transforming how we build software.
The models are learning patterns from data automatically.
Deep learning models use neural networks with many layers.
These learned representations capture complex patterns.
"""

# Step 1: Tokenize (split into words, lowercase, drop punctuation)
tokens = re.findall(r"\b[a-z]+\b", text.lower())
print(f"Tokens ({len(tokens)}): {tokens[:10]}...")

# Step 2: Remove stop words
stop_words = {"the", "is", "are", "how", "we", "from", "with", "many", "these", "a", "an"}
filtered = [t for t in tokens if t not in stop_words]
print(f"After stop words ({len(filtered)}): {filtered[:10]}...")

# Step 3: Stemming (crude but fast: chop common endings)
def simple_stem(word):
    for suffix in ["ing", "ed", "s", "ly", "tion"]:
        if word.endswith(suffix) and len(word) > len(suffix) + 2:
            return word[:-len(suffix)]
    return word

stemmed = [simple_stem(t) for t in filtered]
print(f"After stemming: {stemmed[:10]}...")

# Word frequencies
freq = Counter(stemmed)
print(f"\nTop 5 words: {freq.most_common(5)}")

▶ Output

Tokens (31): ['machine', 'learning', 'is', 'transforming', 'how', 'we', 'build', 'software', 'the', 'models']...
After stop words (22): ['machine', 'learning', 'transforming', 'build', 'software', 'models', 'learning', 'patterns', 'data', 'automatically']...
After stemming: ['machine', 'learn', 'transform', 'build', 'software', 'model', 'learn', 'pattern', 'data', 'automatical']...

Top 5 words: [('learn', 4), ('model', 2), ('pattern', 2), ('machine', 1), ('transform', 1)]

What happened here: The text started as 31 tokens, dropped to 22 after stop word removal, and then stemming folded the variants together so “learning” and “learned” both became “learn”. That is why “learn” tops the count with 4 hits even though no single sentence repeats it that often. Look closely and you will spot the cost of a crude stemmer too: “automatically” became “automatical” because our toy rule blindly chops “ly”. It is ugly but consistent, and consistency is what a vectorizer cares about. A real lemmatizer would do better, which is the exact topic two sections down.

Removing Stop Words

Think of listening to a friend tell a long story. Your brain quietly skips over the “um”, “like”, and “you know” and holds on to the words that carry the plot. Stop word removal is that same instinct written as code. Stop words are the filler of a language: “the”, “is”, “and”, “of”. They glue sentences together but carry almost no meaning on their own, and they appear in nearly every document, so they drown out the words that actually distinguish one review from another. Rather than hand-typing a stop word set like we did above, NLTK ships a curated list for English (and dozens of other languages).

📄 stop_words.py: use NLTK’s curated English list

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_en = set(stopwords.words("english"))
print(f"English stop words available: {len(stop_en)}")
print(f"A few of them: {sorted(stop_en)[:8]}")

review = "The pizza was absolutely delicious and the service was very fast"
tokens = [w.lower() for w in word_tokenize(review) if w.isalpha()]
print(f"\nBefore ({len(tokens)}): {tokens}")

kept = [w for w in tokens if w not in stop_en]
print(f"After  ({len(kept)}): {kept}")

▶ Output

English stop words available: 198
A few of them: ['a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all']

Before (11): ['the', 'pizza', 'was', 'absolutely', 'delicious', 'and', 'the', 'service', 'was', 'very', 'fast']
After  (5): ['pizza', 'absolutely', 'delicious', 'service', 'fast']

What happened here: NLTK gave us 198 English stop words for free. The review shrank from 11 tokens to 5, and look at what survived: “pizza”, “absolutely”, “delicious”, “service”, “fast”. Those five words tell you everything about the sentiment. “the”, “was”, “and”, “very” got cut because they would have appeared in a one-star rant just as often. One caution: stop word lists are blunt. For sentiment work, words like “not” and “no” are sometimes on the list, and removing them can flip “not good” into “good”. Always glance at the list before trusting it on negation-heavy text.

Stemming vs Lemmatization

Stemming and lemmatization are the next stop in NLP basics: both shrink a word to a base form so that “studies” and “studying” count as the same feature. They go about it very differently. Stemming is the chainsaw: it hacks letters off the end using simple rules, fast but often leaving a stub that is not a real word. Lemmatization is the surgeon: it looks the word up in a dictionary (WordNet) and returns the proper root, slower but cleaner. Picture autocorrect on your phone, lemmatization is the version that knows “geese” maps to “goose”, while stemming would just lop off an “e” and shrug.

📄 stem_vs_lemma.py: chainsaw versus surgeon

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

words = ["studies", "studying", "better", "running", "geese", "caring"]

print(f"{'word':<12}{'stem':<12}{'lemma':<12}")
print("-" * 36)
for w in words:
    stem = stemmer.stem(w)
    # pos='v' tells the lemmatizer to treat the word as a verb where it helps
    lemma = lemmatizer.lemmatize(w, pos="v")
    print(f"{w:<12}{stem:<12}{lemma:<12}")

▶ Output

word        stem        lemma
------------------------------------
studies     studi       study
studying    studi       study
better      better      better
running     run         run
geese       gees        geese
caring      care        care

What happened here: Compare the columns. The stemmer turned "studies" and "studying" into "studi", a stub that is not a word but is at least consistent. The lemmatizer turned both into the real word "study". On "geese" the verb lemmatizer left it as "geese" (it only normalizes verbs here, and "geese" is a noun), while the stemmer mangled it into "gees". Neither tool is magic. The rule of thumb: for a quick classifier where the model only needs consistency, stemming is fine and faster. For anything a human will read (search results, chatbot replies), reach for lemmatization so the output stays in real words.

TF-IDF Vectorization

Picture walking into a crowded party. Everyone says "hi" and "how are you", so those words tell you nothing about who is who. But the moment one guest mentions "astrophysics", you instantly learn something specific about them. TF-IDF scores words the same way: the ones everybody uses count for almost nothing, and the rare, distinctive ones count for a lot. This is the step where words finally become numbers. scikit-learn's TfidfVectorizer does the whole job in two method calls: fit learns the vocabulary, and transform scores every document against it. Of all the NLP basics in this post, TF-IDF is the one you will still be reaching for years from now.

The result is a matrix with one row per document and one column per word in the vocabulary. Each cell holds that word's TF-IDF score: high when the word is frequent in this document but rare across the corpus, low when it is everywhere.

📄 tfidf_demo.py: text to feature vectors

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
import numpy as np

documents = [
    "Python is great for machine learning",
    "Machine learning needs lots of data",
    "Data science uses Python and statistics",
    "Deep learning is a subset of machine learning",
    "Statistics helps understand data patterns",
]

# Bag of Words (plain counts)
count_vec = CountVectorizer()
bow = count_vec.fit_transform(documents)

# TF-IDF (importance weighted)
tfidf_vec = TfidfVectorizer()
tfidf = tfidf_vec.fit_transform(documents)

feature_names = tfidf_vec.get_feature_names_out()

print(f"Documents: {len(documents)}")
print(f"Vocabulary size: {len(feature_names)}")
print(f"Feature matrix shape: {tfidf.shape}")

# Show TF-IDF scores for document 0
print(f"\nDocument 0: \"{documents[0]}\"")
doc0_scores = tfidf[0].toarray().flatten()
nonzero = [(feature_names[i], doc0_scores[i]) for i in doc0_scores.nonzero()[0]]
for word, score in sorted(nonzero, key=lambda x: -x[1]):
    print(f"  {word:<15} TF-IDF: {score:.3f}")

▶ Output

Documents: 5
Vocabulary size: 19
Feature matrix shape: (5, 19)

Document 0: "Python is great for machine learning"
  for             TF-IDF: 0.488
  great           TF-IDF: 0.488
  is              TF-IDF: 0.394
  python          TF-IDF: 0.394
  learning        TF-IDF: 0.327
  machine         TF-IDF: 0.327

What happened here: The five documents produced a vocabulary of 19 words, so every document is now a row of 19 numbers, shape (5, 19). For document 0, "for" and "great" tied for the top score (0.488) because each appears in only this one document out of five, making them the most distinctive words here. "python" and "is" scored lower (0.394) since they each show up in 2 of the 5 documents. "learning" and "machine" scored lowest (0.327) because they appear in 3 of the 5, making them common and therefore less useful for telling this document apart.

That is TF-IDF doing its one job: rewarding the words that make a document unique and discounting the words everybody uses. One honest caveat: filler words like “for” and “is” beating “machine learning” looks like the exact opposite of the party analogy. That happens only because this corpus is five tiny sentences, so “for” genuinely is rare here (1 document in 5). On a real corpus of thousands of documents, IDF crushes filler words, and passing stop_words="english" (exactly what the classifier below does) removes them before scoring even starts.

Practical Workflow: A Text Classifier

Now we put all the NLP basics together into the task you actually came for: feed in review text, get back "positive" or "negative". scikit-learn's Pipeline lets you chain the vectorizer and the classifier into a single object, so the exact same text preparation runs at training time and at prediction time. That detail matters more than it looks. If you vectorize your training data one way and your live data another way, your model quietly breaks. A pipeline makes that mistake impossible.

📄 text_classifier.py: TF-IDF plus Naive Bayes in one pipeline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Tiny labelled dataset: 1 = positive review, 0 = negative review
texts = [
    "I love this phone, the battery is amazing",
    "Fantastic camera and a beautiful screen",
    "Best laptop I have ever owned, super fast",
    "The sound quality is excellent and clear",
    "Great value for money, highly recommend",
    "Wonderful experience, the app works perfectly",
    "Smooth performance and a gorgeous display",
    "Comfortable, lightweight and well built",
    "I really love the fast and smooth screen",
    "Amazing value, the best camera I have used",
    "Terrible battery, it dies within an hour",
    "The screen cracked on the first day, awful",
    "Worst purchase ever, a complete waste of money",
    "Slow, buggy and crashes all the time",
    "Poor build quality, it broke after a week",
    "Disappointing camera and bad customer support",
    "Cheap plastic feel and a horrible keyboard",
    "Overpriced and underwhelming, very let down",
    "Awful slow phone that crashes and freezes",
    "Bad battery and a terrible cheap screen",
]
labels = [1] * 10 + [0] * 10

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.3, random_state=42, stratify=labels
)

# Vectorizer plus classifier wired together in one object
pipe = Pipeline([
    ("tfidf", TfidfVectorizer(stop_words="english")),
    ("clf", MultinomialNB()),
])

pipe.fit(X_train, y_train)
pred = pipe.predict(X_test)

print(f"Test samples: {len(y_test)}")
print(f"Test accuracy: {accuracy_score(y_test, pred):.2f}")

# Predict on brand new reviews the model never saw during training
new_reviews = [
    "absolutely love it, fantastic value",
    "broke immediately, a total waste",
]
print()
for review, label in zip(new_reviews, pipe.predict(new_reviews)):
    sentiment = "positive" if label == 1 else "negative"
    print(f"{sentiment:>8}  <-  {review!r}")

▶ Output

Test samples: 6
Test accuracy: 0.83

positive  <-  'absolutely love it, fantastic value'
negative  <-  'broke immediately, a total waste'

What happened here: With random_state=42 pinning the split, the pipeline trained on 14 reviews and was tested on the 6 it had never seen, scoring 0.83 accuracy, so 5 of 6 correct. That is honest for a dataset this tiny, and the two fresh reviews at the bottom were both labelled correctly. Notice how little code it took: one Pipeline, one fit, one predict. In production you would feed in thousands of real reviews instead of twenty, and the accuracy would climb well past this. The shape of the code, though, stays exactly the same.

The Ecosystem

NLTK and scikit-learn cover the NLP basics and make a friendly on-ramp, but they are not the whole road. Here is what to reach for as your needs grow.

• spaCy 3.8.14 is the production NLP workhorse. It is faster than NLTK and bundles a real part-of-speech tagger, named entity recognition, and dependency parsing in one tidy nlp(text) call. When you outgrow tokenizing and stop words, this is the next stop.
• Hugging Face Transformers is where you go when TF-IDF stops being enough. It gives you pretrained models like BERT (Bidirectional Encoder Representations from Transformers) and its many cousins that understand context, so "bank" near "river" and "bank" near "money" get different representations. More power, more compute, usually a GPU.
• sentence-transformers turns whole sentences into dense embedding vectors, perfect for semantic search and clustering where TF-IDF's word-matching falls short.
• scikit-learn's own pieces pair naturally with everything here: Pipeline, GridSearchCV for tuning the vectorizer, and LogisticRegression as a stronger baseline than Naive Bayes once you have more data.

The honest advice: start with the TF-IDF plus simple-classifier combo from this post. It is fast, it runs on a laptop, and it gives you a baseline number. Only reach for transformers when that baseline is not good enough and you can prove the extra accuracy is worth the cost.

Common Mistakes

Mistake 1: Letting the vocabulary explode

On a real corpus the vocabulary can balloon to 100,000 words or more, and most of them are rare typos and one-off names that add noise, not signal. They blow up memory and slow training for nothing. Cap the vocabulary with max_features and let scikit-learn keep only the words that matter.

✅ Correct: cap the vocabulary and drop stop words

from sklearn.feature_extraction.text import TfidfVectorizer

# Keep only the 10,000 most useful words, drop English stop words
tfidf = TfidfVectorizer(max_features=10000, stop_words="english")
print("max_features=10000 keeps the 10K most important words.")
print("stop_words='english' removes common English words automatically.")

▶ Output

max_features=10000 keeps the 10K most important words.
stop_words='english' removes common English words automatically.

Mistake 2: Fitting the vectorizer on the test set

Call fit only on training data. If you fit the vectorizer on the full dataset (train plus test), the vocabulary and the IDF weights secretly learn from your test reviews, and your reported accuracy comes out higher than the model deserves. That is data leakage. The Pipeline in the workflow section prevents it for free, because it only ever calls fit on whatever you pass to pipe.fit().

Mistake 3: Stripping out negation words

For sentiment tasks, blindly removing stop words can delete "not", "no", and "never". Drop "not" from "not good" and the model sees "good" and gets the label backwards. Before you trust a stop word list on review data, check whether it kills your negations, and keep them if it does.

Conclusion and Next Steps

You just walked the entire bridge from raw text to a working classifier. You tokenized sentences and words with NLTK, stripped out stop words, compared stemming against lemmatization, turned words into numbers with TF-IDF, and wired a vectorizer and a Naive Bayes model into a single scikit-learn pipeline that predicts sentiment on reviews it has never seen. The big lesson to carry forward: the model is the easy part, and clean, consistent text preparation is what sets your ceiling. If you want to push that classifier further, the sentiment analysis tutorial builds a fuller pipeline around this exact task.

Next up is sentiment analysis in depth, where you take these NLP basics and push them into a fuller, more accurate pipeline. From there the road leads to word embeddings and transformer models for the day TF-IDF runs out of room. For now, grab a real dataset, swap it into the pipeline from this post, and watch your accuracy climb.

Ready for more? Explore the full Python + AI/ML tutorial series home for every tutorial from first steps to production machine learning.

Frequently Asked Questions

Interview Questions on NLP Basics

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: What is tokenization, and why not just use text.split()?

Tokenization breaks text into meaningful units such as words, punctuation marks, and sentences. Plain text.split() only cuts on whitespace, so "okay." keeps its period glued on and contractions like "don't" are mishandled. NLTK's word_tokenize knows the rules of the language and peels punctuation into its own tokens, which keeps "okay." and "okay" counting as the same word later in the pipeline.

Q: Explain what the IDF part of TF-IDF actually does.

IDF, or inverse document frequency, measures how rare a word is across the whole corpus. A word that appears in every document gets an IDF close to zero, while a word in only a few documents gets a high IDF. Multiplying term frequency by IDF pushes down common filler and rewards distinctive words. It is exactly what stops "the" and "is" from dominating the feature matrix.

Q: Your text classifier scores 0.98 on the test set but performs poorly in production. What do you check first?

Suspect data leakage. The most common cause is fitting the vectorizer on the full dataset before the train/test split, so the vocabulary and IDF weights secretly saw the test rows. Fix it by fitting only on training data, ideally by wrapping the vectorizer and classifier in a Pipeline so fit only ever touches training rows. Also check for duplicate rows that appear in both train and test.

Q: A sentiment model keeps labeling "not good" and "the app was not great" as positive. What is likely wrong, and how do you fix it?

The stop word list is probably stripping negation words like "not", "no", and "never", so "not good" collapses down to just "good". Remove those words from your stop list, or skip stop word removal entirely for sentiment work. Adding bigrams with ngram_range=(1, 2) helps too, because then "not good" becomes a single feature the model can learn as its own signal.

Q: What does the max_features parameter of TfidfVectorizer do, and why set it?

It caps the vocabulary to the top N terms by frequency and discards the rest. On a large corpus the vocabulary can explode past 100,000 terms, most of them rare typos and one-off names that add noise instead of signal. Setting max_features keeps the feature matrix small, training fast, and the signal-to-noise ratio high.

Q: You need to process 10 million documents and NLTK's pipeline is too slow. What do you change?

Move the language steps to spaCy, which is built in Cython and runs far faster, and use nlp.pipe() to process documents in batches. Disable the components you do not need, such as the parser and named entity recognition, if you only want tokens. For vectorization, keep TfidfVectorizer but cap max_features, and consider HashingVectorizer so you never have to hold the entire vocabulary in memory at once.

Series: Python + AI/ML Cookbook. Part 5: Machine Learning

Further reading: the official Python documentation is the authoritative source on this.

Related Posts

Previous: ML: Time Series Analysis with ARIMA and Decomposition

Next: ML: Sentiment Analysis, Bag-of-Words to ML

Series Home: Python + AI/ML Tutorial Series