RNN LSTM models let you process sequential data that a plain feedforward network cannot, by reading one step at a time and remembering what came before. You will understand hidden states, the vanishing gradient problem in sequences, and how LSTM gates fix it, then build a simple RNN and a bidirectional sequence classifier in PyTorch, with every output run on a real machine.

“LSTMs can learn to bridge time intervals in excess of 1000 steps. No other RNN architecture could do this before.”

Sepp Hochreiter, LSTM paper (1997)

Last Updated: July 2026 | Tested on: Python 3.14.6, PyTorch 2.12.1 | Difficulty: Advanced | Reading Time: 11 minutes

Images have spatial structure. Text and time series have something else: order in time. The word “bank” means two completely different things in “river bank” and “bank account”, and the only way to tell which one you mean is the words around it. A plain feedforward network cannot do that, because it looks at each input on its own with no memory of what came before. Recurrent neural networks (RNNs) fix this by keeping a hidden state, a small running summary that gets passed from one step to the next.

Think of reading a long mystery novel. By the last chapter you still remember the clue from chapter one, but only because your brain kept a running summary the whole way through. That summary is the hidden state. At every word, the RNN updates its summary using the new word plus whatever it remembered so far.

The simple RNN has one serious flaw, the vanishing gradient problem on long sequences. When the network learns, an error signal has to travel backward through every step. The further back it goes, the more it shrinks, until it is so close to zero that the early steps barely get updated at all. In plain terms, the network forgets the beginning of a long sequence. Long Short-Term Memory networks (LSTMs) fix this by adding a cell state, a separate memory lane that runs straight through time like a conveyor belt.

Three gates decide what to drop off the belt, what to add to it, and what to read from it. Because the cell state mostly flows along untouched, the learning signal survives across hundreds of steps instead of fading out.

Here is what we cover:

• How RNNs process sequences step by step
• The vanishing gradient problem in long sequences
• LSTM gates: forget, input, output, and cell state
• Building a simple RNN in PyTorch and reading its hidden state
• Sequence classification with a bidirectional LSTM

Prerequisites

How RNN LSTM Models Work

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows how recurrent networks walk through a sequence. At each time step the RNN takes the current input and the previous hidden state, mixes them together, and produces an output plus a fresh hidden state that gets handed to the next step. The LSTM on the right adds the three gates (forget, input, output) that decide what to keep, add, or throw away from the cell state. Those gates are what let an LSTM remember things from far back, which a plain RNN cannot. The two code examples below build each half of the RNN LSTM pair: a simple RNN first, then a bidirectional LSTM classifier.

Simple RNN: Memory Across Time

Think of the hidden state like a single sticky note you keep updating as you read. You never start a fresh note for every word, you just revise the one note so it always reflects everything you have read so far. That one note, once you reach the end, is your summary of the whole sequence.

📄 simple_rnn.py: how the hidden state moves through a sequence

import torch
import torch.nn as nn

# Aditi builds a simple RNN for sequence processing
rnn = nn.RNN(input_size=10, hidden_size=20, num_layers=1, batch_first=True)

# Input: batch of 3 sequences, each 5 steps long, each step has 10 features
x = torch.randn(3, 5, 10)

# Forward pass
output, hidden = rnn(x)

print(f"Input shape:  {x.shape}")          # (batch, seq_len, features)
print(f"Output shape: {output.shape}")      # (batch, seq_len, hidden_size)
print(f"Hidden shape: {hidden.shape}")      # (num_layers, batch, hidden_size)
print(f"\nOutput at last step == Hidden state: {torch.allclose(output[:, -1, :], hidden[0])}")
print("\nThe hidden state at the last step is the sequence representation.")
print("It encodes the entire sequence into a fixed-size vector.")

▶ Output

Input shape:  torch.Size([3, 5, 10])
Output shape: torch.Size([3, 5, 20])
Hidden shape: torch.Size([1, 3, 20])

Output at last step == Hidden state: True

The hidden state at the last step is the sequence representation.
It encodes the entire sequence into a fixed-size vector.

What happened here: The RNN walks through the sequence one step at a time. At each step it combines the current input (10 features) with the previous hidden state (20 values) to make a new hidden state. The output tensor holds the hidden state from all 5 steps, which is handy for sequence-to-sequence work like tagging every word. The final hidden state packs the whole sequence into one 20-dimensional vector, which is what you feed a classifier. The last line confirms it: the output at the final step is exactly the returned hidden state, so torch.allclose prints True.

LSTM Sequence Classifier

📄 lstm_classifier.py: sentiment classification with a bidirectional LSTM

import torch
import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        self.classifier = nn.Sequential(
            nn.Linear(hidden_dim * 2, 64),  # *2 for bidirectional
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(64, num_classes),
        )

    def forward(self, x):
        embedded = self.embedding(x)
        lstm_out, (hidden, cell) = self.lstm(embedded)
        # Concatenate forward and backward final hidden states
        combined = torch.cat([hidden[0], hidden[1]], dim=1)
        return self.classifier(combined)

# Anvay tests the architecture
model = LSTMClassifier(vocab_size=10000, embed_dim=128, hidden_dim=64, num_classes=2)
x_sample = torch.randint(0, 10000, (4, 50))  # 4 sentences, 50 tokens each
output = model(x_sample)
print(f"Input: {x_sample.shape} (batch=4, seq_len=50)")
print(f"Output: {output.shape} (batch=4, classes=2)")

params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {params:,}")
print("\nArchitecture: Embedding -> Bidirectional LSTM -> FC Classifier")
print("Bidirectional reads the sequence both forward and backward.")

▶ Output

Input: torch.Size([4, 50]) (batch=4, seq_len=50)
Output: torch.Size([4, 2]) (batch=4, classes=2)
Total parameters: 1,387,714

Architecture: Embedding -> Bidirectional LSTM -> FC Classifier
Bidirectional reads the sequence both forward and backward.

What happened here: The embedding layer turns token IDs into dense vectors (10000 words, each mapped to a 128-number vector). The bidirectional LSTM reads the sentence both ways at once: one pass left to right (what came before) and one pass right to left (what comes after). Reading a review backward as well as forward is like proofreading a sentence twice from both ends, you catch context you would miss in a single pass. The two final hidden states get stitched together into a 128-number vector (64 from the forward pass, 64 from the backward pass) and handed to the classifier. The model has 1,387,714 parameters, and almost all of them (1,280,000) live in the embedding table, not the LSTM.

Common Mistakes

Interview Corner

Q: How does LSTM solve the vanishing gradient problem?

The cell state acts like an express lane for the learning signal. It moves through time using only simple element-wise operations (add and multiply), not the big matrix multiplications that shrink the signal step after step in a plain RNN. The forget gate decides what to drop from memory, the input gate decides what new information to write in, and the output gate decides what to read out. Because the cell state can travel many steps almost untouched, the gradient flows back to the early steps without fading to zero. Picture a parcel riding a conveyor belt past every station: it can move a long way without anyone repacking it.

Q: LSTM vs GRU, when do you pick which?

GRU merges the forget and input gates into a single update gate, making it simpler (fewer parameters, faster training). Performance is similar for most tasks. Use LSTM as default; switch to GRU if you need faster training or have limited memory. For most Natural Language Processing (NLP) tasks in 2026, Transformers have replaced both.

Practice Exercises

1. Build a character-level text generator using an LSTM trained on Shakespeare text.
2. Compare LSTM, GRU, and simple RNN on the same classification task. Measure accuracy and training time.
3. Implement a stacked LSTM (2-3 layers) and compare with a single-layer LSTM.
4. Train a bidirectional LSTM for named entity recognition (sequence labeling task).

More in this series:

• DL: Transfer Learning with ResNet, VGG Pre-trained Models
• Deep Learning Capstone: Image Classifier on Hugging Face
• The Attention Mechanism From Scratch in PyTorch

Frequently Asked Questions

Interview Questions on RNNs and LSTMs

How interviewers actually probe this topic: real scenarios, with answers you can say out loud.

Q: Why can a plain feedforward network not handle text or time series, but an RNN can?

A feedforward network looks at each input on its own with no memory of what came before, so it cannot use order, and in language order changes meaning (“river bank” versus “bank account”). An RNN adds a hidden state that is passed from one step to the next, giving the network a running summary of everything it has seen so far. That memory is what lets it model sequences.

Q: What is the difference between the output tensor and the hidden state returned by nn.RNN or nn.LSTM?

The output tensor holds the hidden state from every time step, shape (batch, seq_len, hidden_size) with batch_first=True. The returned hidden holds only the last step’s hidden state for each layer, shape (num_layers, batch, hidden_size). For a single-layer unidirectional RNN, output[:, -1, :] equals hidden[0]. Use output for per-token tasks like tagging, and hidden for whole-sequence classification.

Q: What does the cell state do that the hidden state does not?

The cell state is the LSTM’s long-term memory lane that flows through time using only element-wise add and multiply, so information and gradients can survive many steps. The hidden state is the short-term, filtered output read out through the output gate at each step. The gates edit the cell state, and the hidden state is what the rest of the network actually sees.

Q: Why does a bidirectional LSTM double the classifier’s input size?

A bidirectional layer runs two independent LSTMs, one left to right and one right to left, then concatenates their hidden states, so the final representation has 2 * hidden_dim features. That is why the classifier’s first Linear layer uses hidden_dim * 2 as its input, as in the example above. Bidirectional only works when the full sequence is available up front, not for streaming text generation.

Q: Your bidirectional LSTM trains fine on fixed-length batches, but once you feed variable-length reviews the predictions look random. What do you check first?

The usual culprit is padding tokens corrupting the hidden state. Without pack_padded_sequence(), the LSTM processes padding as real input, so the backward pass of a bidirectional model starts its “final” state from padding rather than from the last real token. Pass the true sequence lengths and pack the batch so padding is skipped. Also confirm batch_first matches your tensor layout of (batch, seq_len, features).

Q: Your LSTM model trains but Graphics Processing Unit (GPU) memory spikes on certain batches and throughput is uneven. What is likely happening?

If you pad every batch to the length of the longest sequence in the whole dataset, one very long sample blows up memory and wastes compute on padding. Bucket sequences of similar length together and pad per batch, and use pack_padded_sequence() so the LSTM does not process padding. If you carry hidden states across batches, detach them so the computation graph does not grow and pin extra memory.

What’s Next?

You now understand how RNNs carry a hidden state across time, why plain RNNs forget long sequences because of the vanishing gradient problem, and how the LSTM cell state and its three gates keep memory alive across hundreds of steps. You also built a simple RNN and read its hidden state, then wired up a bidirectional LSTM sequence classifier in PyTorch and checked every shape and parameter count on a real machine. That is the whole RNN LSTM story: state in, memory managed by gates, prediction out.

An RNN LSTM stack processes sequences, but it still treats each word as a plain index number with no meaning of its own. In the word embeddings tutorial, we learn how to turn words into dense vectors that actually capture meaning, the trick where “king minus man plus woman equals queen” works out as real vector arithmetic.

Want the full path from basics to advanced AI/ML? Explore the complete Python + AI/ML tutorial series home for every tutorial in order.

Further reading: for the full reference, see the official Python documentation.

Related Posts

Previous: NLP: Word Embeddings (Word2Vec, GloVe, FastText)

Next: Deep Learning Capstone: Image Classifier on Hugging Face

Series Home: Python + AI/ML Tutorial Series