Word embeddings turn words into vectors that capture meaning. Train Word2Vec from scratch with Gensim, explore GloVe pre-trained embeddings, and see how FastText handles words it has never met before. “King minus Man plus Woman equals Queen” explained with real, tested code in this word2vec python guide.
“You shall know a word by the company it keeps.”
J.R. Firth, linguist
Last Updated: July 2026 | Tested on: Gensim 4.4.0 on Python 3.13 (no 3.14 wheels yet) | Difficulty: Advanced | Reading Time: 14 minutes
How does a computer know that “dog” and “puppy” are related, but “dog” and “quantum” are not? On its own, it does not. A computer only sees numbers, so we have to hand it numbers that carry meaning. The old approach, one-hot encoding (see the NLP basics tutorial), gives every word its own column and treats all words as equally far apart. By that measure “cat” is exactly as close to “kitten” as it is to “helicopter”, which is useless. Word embeddings fix this by mapping each word to a short list of numbers, called a dense vector (usually 100 to 300 numbers long), placed so that words used in similar ways sit close together.
Here is the everyday version. Think of a seating chart at a big wedding. You do not know the guests, but you can guess relationships just from where people sit. The folks at table 4 probably went to college together. The crowd near the bar probably works at the same office. Word embeddings build that seating chart for an entire language: every word gets a seat, and words that keep showing up in the same company end up at the same table. That is the whole idea, and it is exactly what the linguist J.R. Firth meant by “you shall know a word by the company it keeps”.
The breakthrough came in 2013, when Tomas Mikolov and his team at Google published Word2Vec. The idea sounds almost too simple to work: train a small neural network to predict a word from its neighbours (or the neighbours from the word), then throw the predictions away and keep the weights it learned. Those weights become the word vectors. Words that show up in similar sentences end up with similar vectors, and something surprising falls out for free.
The vector for “king” minus “man” plus “woman” lands right next to “queen”. GloVe (Stanford, 2014) reaches a similar place by crunching a giant table of how often words appear near each other. FastText (Facebook, 2016) builds on Word2Vec by also learning pieces of words, so it can handle typos and words it has never seen.
Here is what we cover:
- Why one-hot encoding fails for NLP (Natural Language Processing)
- Word2Vec: Skip-gram and CBOW architectures
- Training embeddings with Gensim
- Word analogies and similarity queries
- GloVe and FastText comparisons
Table of Contents
Prerequisites
The diagram traces the journey. Raw text goes in, gets split into tokens, and each token gets turned into a dense vector by the embedding layer. The result is a vector space where similar words sit close together. We start with a sparse vocabulary space (one-hot vectors with thousands of dimensions, mostly zeros) and squeeze it down to a dense space (usually 100 to 300 numbers per word). The classic example, king minus man plus woman lands near queen, shows that meaning becomes geometry: a relationship turns into a direction you can add and subtract. Word2Vec, GloVe, and FastText each learn these vectors a different way, but the spaces they produce behave alike.
- RNN and LSTM tutorial
- NLP basics tutorial
pip install gensim==4.4.0- Heads up: at the time of writing Gensim ships wheels for Python 3.9 to 3.13 only. There is no Python 3.14.6 build yet, so the code here was tested on Python 3.13. Run these examples on a 3.13 environment for now.
Training Word2Vec with Gensim
Word2Vec comes in two flavours, and the difference is just which direction you guess. Think of the fill-in-the-blank questions from a school grammar test: CBOW is handed a sentence with one word blanked out and guesses the missing word, while Skip-gram is handed that one word and guesses the blanks that surround it. Skip-gram looks at one centre word and tries to predict the words around it. It tends to do better on rare words and smaller datasets.
CBOW (Continuous Bag of Words) does the opposite: it looks at the surrounding words and tries to guess the word in the middle. CBOW trains faster and does well on common words. Both end up producing the same kind of dense vectors; only the guessing game changes. The vectors that come out capture grammar patterns (verb tenses, plurals) and meaning patterns (country to capital, male to female) as straight-line directions you can travel along.
📄 word2vec_training.py: training word embeddings on a tiny corpus
from gensim.models import Word2Vec
import numpy as np
# Aditi trains Word2Vec on sample text
sentences = [
["python", "is", "a", "programming", "language"],
["java", "is", "a", "programming", "language"],
["javascript", "is", "a", "programming", "language"],
["python", "has", "great", "machine", "learning", "libraries"],
["pytorch", "and", "tensorflow", "are", "deep", "learning", "frameworks"],
["numpy", "and", "pandas", "are", "data", "science", "libraries"],
["python", "is", "used", "for", "data", "science"],
["java", "is", "used", "for", "enterprise", "applications"],
["javascript", "is", "used", "for", "web", "development"],
["machine", "learning", "uses", "neural", "networks"],
["deep", "learning", "is", "a", "subset", "of", "machine", "learning"],
["data", "science", "combines", "statistics", "and", "programming"],
]
# Train model (small corpus for demo)
# seed=42 + workers=1 makes the run fully reproducible
model = Word2Vec(sentences, vector_size=50, window=3, min_count=1,
epochs=100, seed=42, workers=1)
print("Vocabulary size:", len(model.wv))
print(f"Embedding shape for 'python': {model.wv['python'].shape}")
print("\nMost similar to 'python':")
for word, score in model.wv.most_similar("python", topn=5):
print(f" {word:>15}: {score:.4f}")
print("\nMost similar to 'learning':")
for word, score in model.wv.most_similar("learning", topn=5):
print(f" {word:>15}: {score:.4f}")
▶ Output
Vocabulary size: 35
Embedding shape for 'python': (50,)
Most similar to 'python':
deep: 0.3246
machine: 0.2141
for: 0.2073
numpy: 0.2029
of: 0.1617
Most similar to 'learning':
deep: 0.4211
libraries: 0.3326
combines: 0.3144
a: 0.2995
science: 0.2301
What happened here: The code ran, the model trained, and the shapes are right: 35 unique words in the vocabulary, and each word is a point in 50-dimensional space (that is the vector_size=50). But look at the actual similarity scores. They are low and a bit random. The top match for “python” is “deep” at 0.32, and filler words like “for”, “of”, and “a” sneak into the top five. The neat story you might expect, that python, java, and javascript would be best friends, simply does not show up.
Twelve sentences is nowhere near enough text. Word2Vec needs to see a word in many different contexts before its vector settles into a meaningful spot, and a dozen short lines do not give it that. This is the single most common beginner trap with embeddings, so it gets its own section next. The fix is not a bigger neural network. It is more text, or borrowing vectors someone already trained on billions of words.
Word Analogies: King minus Man plus Woman = Queen
So our tiny model flopped. The honest fix is to stand on the shoulders of giants and load vectors that someone else already trained on billions of words. Gensim makes this a one-liner. Below, a developer named Niranjan downloads GloVe vectors built from Wikipedia plus the Gigaword news corpus: 400,000 words, each described by 50 numbers. Then comes the famous party trick. We do arithmetic on words. Take the vector for “king”, subtract “man”, add “woman”, and ask which word lands nearest. If embeddings really capture meaning, the answer should be “queen”.
📄 word_analogies.py: doing arithmetic on words with pre-trained vectors
import gensim.downloader as api
# Niranjan loads pre-trained GloVe vectors (400K words, 50 dims)
print("Loading pre-trained GloVe embeddings...")
glove = api.load("glove-wiki-gigaword-50")
print(f"Vocabulary: {len(glove):,} words, {glove.vector_size} dimensions")
# Classic analogy: king - man + woman = ?
result = glove.most_similar(positive=["king", "woman"], negative=["man"], topn=3)
print("\nking - man + woman = ?")
for word, score in result:
print(f" {word}: {score:.4f}")
# More analogies
analogies = [
(["paris", "germany"], ["france"], "paris - france + germany = ?"),
(["walking", "swam"], ["walked"], "walking - walked + swam = ?"),
(["bigger", "cold"], ["big"], "bigger - big + cold = ?"),
]
for pos, neg, label in analogies:
result = glove.most_similar(positive=pos, negative=neg, topn=1)
print(f"{label} {result[0][0]} ({result[0][1]:.4f})")
▶ Output
Loading pre-trained GloVe embeddings... Vocabulary: 400,000 words, 50 dimensions king - man + woman = ? queen: 0.8524 throne: 0.7664 prince: 0.7592 paris - france + germany = ? berlin (0.9204) walking - walked + swam = ? swim (0.8120) bigger - big + cold = ? warmer (0.7481)
What happened here: This is the same code as before, but trained on billions of words instead of twelve sentences, and the difference is night and day. “king” minus “man” plus “woman” really does land on “queen” (0.8524), with “throne” and “prince” close behind, which makes sense since they all live in the royalty neighbourhood. The capital-city analogy is even cleaner: “paris” minus “france” plus “germany” gives “berlin” at 0.92. The grammar analogy works too, turning the past tense “swam” into the gerund “swim”.
Notice the last line is honestly a little off: “bigger” minus “big” plus “cold” returns “warmer”, not “colder”. The model nailed the “comparative form” direction (it gave us an -er word) but slid to the opposite temperature, because “warmer” and “colder” show up in nearly identical sentences about weather. That is a good reminder: these analogies are strong tendencies, not guarantees. Nobody hard-coded any of this. The directions emerged on their own, purely from which words keep each other company across a huge pile of text.
Word2Vec vs GloVe vs FastText
These three are the classic static embeddings, and they mostly differ in how they learn and what they do with strange words. The one feature worth calling out is FastText. It learns the small chunks inside a word, like py, pyth, and thon inside “python”. So when it later meets a typo such as “pyhton”, it can still stitch together a decent vector from the pieces it knows. Think of it like recognising a friend from the back: you cannot see the face, but the haircut, jacket, and walk are enough. Word2Vec and GloVe have no such trick. Hand them a word they never saw in training and they simply shrug. Here is the side-by-side.
| Feature | Word2Vec | GloVe | FastText |
|---|---|---|---|
| Training | Predict context words | Co-occurrence matrix | Subword n-grams |
| OOV (Out of Vocabulary) Words | Cannot handle | Cannot handle | Handles via subwords |
| Speed | Fast training | Moderate | Slower (more params) |
| Best For | General NLP | Semantic tasks | Morphological languages |
| Limitation | Static (one vector per word) | Static | Static |
Common Mistakes
- Training Word2Vec on too little text: you saw this happen live up top. Twelve sentences gave us noisy, near-meaningless vectors. Good embeddings need millions of words. For a small corpus, load pre-trained vectors instead of training your own.
- Expecting one vector to cover every meaning of a word: “Apple” the fruit and “Apple” the company share a single static vector. When the exact sense matters, reach for contextual embeddings (BERT) instead.
- Comparing vectors with the wrong measure: use cosine similarity, not raw Euclidean distance. Cosine looks at the direction a vector points and ignores its length, which is exactly what you want for words. Gensim’s
most_similaralready uses cosine for you.
Interview Corner
Q: What is the difference between static and contextual embeddings?
Static embeddings (Word2Vec, GloVe, FastText) assign one fixed vector per word regardless of context. Contextual embeddings (BERT, GPT) generate a different vector for the same word depending on surrounding text. “I went to the bank” and “the river bank” produce different vectors for “bank” with BERT but identical vectors with Word2Vec.
Practice Exercises
- Train Word2Vec on a larger corpus (Wikipedia dump or 20 Newsgroups) and test analogies.
- Compare Word2Vec, GloVe, and FastText similarity scores on the same 20 word pairs.
- Use pre-trained embeddings as input to an LSTM classifier (from RNN and LSTM tutorial) and compare with random initialization.
- Test FastText on deliberately misspelled words. How well does it handle “pyhton” vs “python”?
More in this series:
- AI: Computer Vision Project, Object Detection with YOLO
- The Attention Mechanism From Scratch in PyTorch
- NLP: Transformer Architecture, Attention Is All You Need
Frequently Asked Questions
How do I train Word2Vec in Python?
The easiest way to train Word2Vec in Python is Gensim. Install it with pip install gensim, pass a list of tokenized sentences to Word2Vec(sentences, vector_size=100, window=5, min_count=5), and read the learned vectors from model.wv. For reproducible results set seed=42 and workers=1. Just remember that a tiny corpus gives weak vectors; you need millions of words, or load a pre-trained model.
What embedding dimension should I use?
50 to 100 for small vocabularies or fast prototyping. 200 to 300 for production NLP. The original Word2Vec work popularized 300 dimensions. Going beyond 300 rarely helps and costs more compute. Modern Transformers use 768 or more because they learn contextual embeddings jointly with the model.
Should I use pre-trained embeddings or train my own?
Use pre-trained vectors (GloVe or FastText) unless your domain has very specialized vocabulary such as medical, legal, or scientific text. Pre-trained embeddings carry general language knowledge learned from billions of words. Fine-tune them on your own task for the best results.
Does Gensim work on Python 3.14.6?
At the time of writing, Gensim 4.4.0 ships pre-built wheels for Python 3.9 through 3.13 only, with no Python 3.14.6 wheel yet. Installing on 3.14 tries to compile from source and needs a C++ build toolchain. The simplest fix today is to run your Word2Vec code on Python 3.13. Check the Gensim PyPI page for a 3.14 wheel before assuming it is unavailable.
Interview Questions on Word Embeddings
These come from real screens and onsites. Practice answering before you read each answer.
Q: When would you pick Skip-gram over CBOW, and why?
Pick Skip-gram when your corpus is small or your vocabulary has many rare words, because it treats each centre-and-context pair as a separate training example and so squeezes more signal out of infrequent words. CBOW averages the context together and trains faster, which makes it a better fit for large corpora and common words where speed matters more than rare-word quality. In Gensim you switch between them with sg=1 for Skip-gram and sg=0 for CBOW.
Q: How does FastText produce a vector for a word it never saw during training?
FastText represents each word as a bag of character n-grams, for example the subwords py, pyt, and thon inside “python”. An unseen word or typo still shares many of those subwords with known words, so FastText sums the subword vectors to build a reasonable embedding on the fly. That is why a misspelling like “pyhton” lands close to “python”, whereas Word2Vec and GloVe simply raise a key error for anything outside their vocabulary.
Q: Why do we compare word vectors with cosine similarity instead of Euclidean distance?
Cosine similarity measures the angle between two vectors and ignores their length, so it captures whether two words point in the same semantic direction regardless of how often each word appeared. Euclidean distance mixes in magnitude, which is heavily influenced by word frequency and vector norm, so it can call a frequent word and a rare word far apart even when they mean similar things. Gensim’s most_similar uses cosine for exactly this reason.
Q: You train Word2Vec on your own text and the nearest neighbours for every word are filler words like “the”, “of”, and “a”. What do you check first?
The first suspect is corpus size: a few dozen or a few thousand sentences is far too little, and the vectors never settle, which is exactly the noisy result shown earlier in this post. Next, check that stopwords are handled sensibly and that min_count is not so low that rare noise words survive, and confirm you ran enough epochs. If gathering millions of words is not practical, the real fix is to load pre-trained GloVe or FastText vectors instead of training from scratch.
Q: Your production search handles user-typed queries full of domain jargon and typos, and pre-trained GloVe keeps missing those words. Which embedding do you reach for and why?
Reach for FastText, because its subword n-grams give it a fighting chance on typos and never-before-seen jargon that would be out-of-vocabulary for Word2Vec or GloVe. If the domain vocabulary is very specialized, such as medical or legal text, train or fine-tune FastText on your own corpus so the subword pieces reflect your terminology. For the highest quality on ambiguous queries you would then consider contextual models like BERT, which give a word a different vector per sentence.
Q: What does “king minus man plus woman equals queen” actually demonstrate, and why is it not guaranteed?
It shows that relationships between words become consistent directions in the vector space, so gender or royalty can be added and subtracted like arithmetic, and that structure emerges purely from co-occurrence, not from any hand-coded rule. It is not guaranteed because the directions are strong statistical tendencies rather than exact laws. As the post showed, “bigger minus big plus cold” returned “warmer” instead of “colder”, since words that appear in near-identical sentences can sit closer than the logically correct answer.
What’s Next?
You learned why one-hot encoding falls flat, how Word2Vec turns the company a word keeps into geometry, why a tiny corpus gives noisy vectors while pre-trained GloVe nails “king minus man plus woman equals queen”, and how FastText stitches together vectors for words it has never seen. Word embeddings give words meaning, but the meaning is frozen: one vector per word, no matter the sentence. That is their ceiling. In the transformer architecture tutorial, we dig into the design that changed NLP by reading the whole sentence first, so a model can tell that “bank” in “river bank” is a different thing from “bank account”. Same word, two vectors, picked on the fly.
Want the full roadmap from Python basics to deep learning? Head back to the Python + AI/ML tutorial series home for every lesson in order.
Go deeper: the official Python documentation covers every edge case of this topic.
Related Posts
Previous: AI: Computer Vision Project, Object Detection with YOLO
Next: DL: RNNs and LSTMs for Sequence Processing
Series Home: Python + AI/ML Tutorial Series
No comment