Learn the python generator yield pattern. A python generator is a function that produces values lazily with yield. Understand generator expressions, pipelines, send(), and why generators use almost no memory compared to lists.

“Make it work, make it right, make it fast. In that order.”

Kent Beck, Extreme Programming

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Intermediate | Reading Time: 16 minutes

Think of a Netflix show. You do not download the whole season before you can watch the first minute. The video streams one chunk at a time, and the rest stays on the server until you ask for it. A python generator works the same way. It hands you one value, pauses, and waits. It never builds the whole list up front. That single idea is what makes generators so light on memory.

In the iterators tutorial we built iterators with __iter__ and __next__. That took two methods, a class, manual state tracking, and raising StopIteration by hand. Generators do the same job in three lines. You write yield instead of return, and Python handles the rest: it pauses the function, saves every local variable, and resumes exactly where it left off the next time you ask for a value.

So what do you reach for when you need to process a million items but cannot fit them all in memory at once? A Python generator. Generators are the building blocks of efficient data pipelines, infinite sequences, and the whole itertools module. This post teaches you how they work from the ground up, one tested example at a time.

Your First Generator

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows a generator’s lifecycle through four states: created (when you call the generator function), suspended (after each yield), running (when next() resumes execution), and closed (when the function returns or you close it). The back and forth between “suspended” and “running” is what keeps generators memory efficient. They hand you one value at a time instead of building an entire list up front. That lazy evaluation pattern is exactly what you want when a dataset is too big to hold in memory.

📄 first_generator.py: yield vs return

def countdown(n):
    """Generator that counts down from n to 1."""
    print(f"Starting countdown from {n}")
    while n > 0:
        yield n         # Pause here, produce n
        n -= 1          # Resume here on next call
    print("Countdown finished!")

# Calling the function does not execute it; it returns a generator object
gen = countdown(5)
print(f"Type: {type(gen)}")

# next() resumes execution until the next yield
print(f"First:  {next(gen)}")
print(f"Second: {next(gen)}")
print(f"Third:  {next(gen)}")

# Or use it in a for loop
print("\nFull loop from 3:")
for num in countdown(3):
    print(f"  {num}")

▶ Output

Type: <class 'generator'>
Starting countdown from 5
First:  5
Second: 4
Third:  3

Full loop from 3:
Starting countdown from 3
  3
  2
  1
Countdown finished!

What happened here: Calling countdown(5) does not run the function body. It returns a generator object, frozen at the very top. The first next() call runs until it hits yield n, returns 5, and freezes again right there. The second next() resumes on the line after the yield, decrements n, loops back around, and yields 4. Notice that “Starting countdown from 5” only prints on that first next(), not when you create the generator. When the function finally finishes (falls off the end or hits return), Python raises StopIteration for you. No manual state tracking, no class boilerplate, just a function that remembers where it paused.

Why Generators Matter: Memory

Here is the payoff. Think of a list as photocopying an entire 1,000 page book before you read a single word. A generator is a bookmark: the next page appears only when you turn to it. A list packs every value into memory the moment you build it. A Python generator holds almost nothing: just the code and a bookmark for where it paused. The bigger the data, the more this matters. Let us measure it with one million squared numbers and compare the two side by side.

📄 memory_comparison.py: list vs generator memory usage

import sys

# List: stores ALL values in memory at once
big_list = [x * x for x in range(1_000_000)]
print(f"List size: {sys.getsizeof(big_list):,} bytes ({sys.getsizeof(big_list) // 1024:,} KB)")

# Generator: produces values one at a time
big_gen = (x * x for x in range(1_000_000))
print(f"Generator size: {sys.getsizeof(big_gen)} bytes")

# Same result, vastly different memory
print(f"\nFirst 5 from generator: {[next(big_gen) for _ in range(5)]}")
print(f"Sum from list: {sum(big_list):,}")

# Generators shine when you process data that does not fit in memory:
# - Reading a 50GB log file line by line
# - Processing API responses page by page
# - Generating infinite sequences

▶ Output

List size: 8,448,728 bytes (8,250 KB)
Generator size: 208 bytes

First 5 from generator: [0, 1, 4, 9, 16]
Sum from list: 333,332,833,333,500,000

What happened here: The list grabs about 8 MB to hold all one million squared values at once. The generator uses 208 bytes, full stop. It stores only the code and the current position, then produces each value on demand. That is roughly a 40,000 to 1 difference, and the gap only grows as the data gets bigger. If you only need one item at a time (summing, filtering, or writing each value to a file), a generator saves a massive amount of memory for free.

Generator Expressions: One-Liner Generators

You do not always need a full def with yield. For simple cases there is a one-liner: a generator expression. It looks exactly like a list comprehension, but you swap the square brackets for round parentheses. Think of a restaurant: brackets order the entire thali up front, parentheses ask the kitchen to send one dish only when you are ready for it. Brackets build the whole list right away. Parentheses give you a lazy generator that produces values only when asked. To make it concrete, say you track the ages of a five person dev team: Rahul, Niranjan, Viraj, Pravin, and Anvi.

📄 gen_expressions.py: parentheses instead of brackets

# List comprehension: creates a list (all values in memory)
squares_list = [x**2 for x in range(10)]

# Generator expression: creates a generator (lazy, one at a time)
squares_gen = (x**2 for x in range(10))

print(f"List: {squares_list}")
print(f"Generator: {squares_gen}")  # Not the values, just a generator object

# Generator expressions work directly in functions
team_ages = {"Rahul": 28, "Niranjan": 26, "Viraj": 25, "Pravin": 31, "Anvi": 29}

# sum() accepts a generator, so no intermediate list is needed
total_age = sum(age for age in team_ages.values())
avg_age = total_age / len(team_ages)
print(f"\nTotal age: {total_age}")
print(f"Average age: {avg_age:.1f}")

# any() and all() also accept generators
has_senior = any(age > 30 for age in team_ages.values())
all_young = all(age < 35 for age in team_ages.values())
print(f"Has someone over 30? {has_senior}")
print(f"Everyone under 35? {all_young}")

# min() with a key finds the youngest by age value
youngest = min(team_ages, key=team_ages.get)
print(f"Youngest: {youngest} (age {team_ages[youngest]})")

▶ Output

List: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Generator: <generator object <genexpr> at 0x7f2a1c3d5e40>

Total age: 139
Average age: 27.8
Has someone over 30? True
Everyone under 35? True
Youngest: Viraj (age 25)

What happened here: Printing the generator shows a generator object, not the numbers, because the values do not exist yet. The real win is the second half. Functions like sum(), any(), all(), min(), and max() happily accept a generator expression and pull values straight through it. No intermediate list is built, so you get the clean comprehension syntax with none of the memory cost. The address in the generator line (0x7f2a1c3d5e40) is just a memory location, so yours will look different every run.

Generator Pipelines: Chaining Generators

Generators really shine when you connect them. Picture an assembly line in a factory: each station does one small job and passes the part to the next station. One generator splits text into lines, the next drops the blank lines, the next parses each line, and the last keeps only the high scores. An item moves through every station before the next item even starts. Nothing piles up in the middle. Say a trainer named Aditi has a messy score sheet from her Python batch and only wants the students who scored 80 or more. Here is that exact pipeline.

📄 pipeline.py: process data in stages without loading it all

def read_lines(text):
    """Stage 1: Split text into lines."""
    for line in text.strip().split("\n"):
        yield line

def filter_non_empty(lines):
    """Stage 2: Skip blank lines."""
    for line in lines:
        if line.strip():
            yield line

def parse_scores(lines):
    """Stage 3: Parse 'name:score' format."""
    for line in lines:
        name, score = line.split(":")
        yield {"name": name.strip(), "score": int(score.strip())}

def above_threshold(records, threshold=80):
    """Stage 4: Filter by minimum score."""
    for record in records:
        if record["score"] >= threshold:
            yield record

# Raw data
raw = """
Rahul: 92
Niranjan: 78

Viraj: 95
Prathamesh: 85

Vinay: 67
Pravin: 91
"""

# Build the pipeline: nothing executes yet!
lines = read_lines(raw)
non_empty = filter_non_empty(lines)
records = parse_scores(non_empty)
passing = above_threshold(records, threshold=80)

# Pull values through the pipeline
print("Students scoring 80+:")
for student in passing:
    print(f"  {student['name']}: {student['score']}")

▶ Output

Students scoring 80+:
  Rahul: 92
  Viraj: 95
  Prathamesh: 85
  Pravin: 91

What happened here: Each generator function is one station on the line. When you wire the four stages together, nothing runs yet, because generators are lazy. Only when the for loop asks for a value does data start to flow. Each next() on passing pulls a next() from records, which pulls from non_empty, which pulls from lines. One record travels the whole pipeline, gets printed, and only then does the next record start. For a 50GB log file, just one line sits in memory at any moment, no matter how huge the file is.

Advanced: send() and throw()

So far values have flowed one way: out of the generator. But yield can also receive a value coming back in. With gen.send(value) you push a value into the paused yield, and the expression value = yield total evaluates to whatever you sent. Think of a walkie-talkie instead of a one-way radio: the generator talks, then listens, then talks again. Its sibling gen.throw(exc) pushes an exception instead of a value: Python raises exc right at the paused yield, and the generator can catch it with try/except to clean up or recover, or let it propagate out to you.

📄 send_example.py: two-way communication with generators

def accumulator(initial=0):
    """Generator that accumulates values sent to it."""
    total = initial
    while True:
        value = yield total  # Yield current total, receive new value
        if value is None:
            break
        total += value

acc = accumulator(0)
next(acc)                # Prime the generator (advance to first yield)

print(acc.send(10))      # Send 10, get total: 10
print(acc.send(25))      # Send 25, get total: 35
print(acc.send(7))       # Send 7, get total: 42

# send() injects a value INTO the yield expression.
# The yield expression evaluates to whatever was sent.

▶ Output

10
35
42

What happened here: The running total starts at 0. Each send() drops a number into the waiting yield, the generator adds it to total, loops back, and yields the new total straight back to you. You rarely write send() in everyday code, but it powers coroutines and async frameworks under the hood. One catch: that first next(acc) is required to “prime” the generator, that is, to run it up to the first yield so it is ready to receive. Skip the priming step and Python raises a TypeError, which we hit on purpose in the Common Mistakes section below.

yield from: Delegating to Sub-Generators

When one generator needs to hand off to another, yield from does it cleanly. Instead of writing a for loop that yields each item one by one, you write yield from other_generator() and Python forwards every value for you. It is like a manager who simply says “go ask that team” instead of relaying each answer themselves. In the example below, a full stack developer named Aviraj keeps his frontend and backend skills in two separate generators, and one combined generator serves them all.

📄 yield_from.py: flattening nested generators

def frontend_skills():
    yield "HTML"
    yield "CSS"
    yield "JavaScript"

def backend_skills():
    yield "Python"
    yield "PostgreSQL"
    yield "Docker"

# Without yield from: manual forwarding
def all_skills_manual():
    for skill in frontend_skills():
        yield skill
    for skill in backend_skills():
        yield skill

# With yield from: delegates directly
def all_skills():
    yield from frontend_skills()
    yield from backend_skills()

print("Aviraj's skills:")
for skill in all_skills():
    print(f"  - {skill}")

▶ Output

Aviraj's skills:
  - HTML
  - CSS
  - JavaScript
  - Python
  - PostgreSQL
  - Docker

What happened here: Both all_skills_manual() and all_skills() produce the same six skills, but the yield from version is shorter and easier to read. yield from delegates to another iterable and forwards each value automatically. It also forwards send() and throw() calls correctly, which matters once you start building coroutines. For everyday code, treat it as the clean way to chain or flatten generators.

The Catch: Generators Are Single-Use

Here is the one thing that bites almost everyone. A Python generator is single use, like a movie ticket. You walk through the gate once and the ticket is spent. Once a generator runs out of values, it is exhausted for good. Loop over it a second time and you get nothing, no error, just silence. That silent empty result is what makes this bug so sneaky.

📄 single_use.py: the trap that catches everyone

def team_members():
    yield "Rahul"
    yield "Niranjan"
    yield "Viraj"

gen = team_members()

# First pass works
print("First pass:", list(gen))   # ['Rahul', 'Niranjan', 'Viraj']

# Second pass is EMPTY: the generator is exhausted!
print("Second pass:", list(gen))  # []

# Fix: call the generator function again to get a fresh generator
print("Fresh:", list(team_members()))  # ['Rahul', 'Niranjan', 'Viraj']

# Or: if you need multiple passes, convert to a list first
team_list = list(team_members())
print("Reusable:", team_list)     # Use the list as many times as you want

▶ Output

First pass: ['Rahul', 'Niranjan', 'Viraj']
Second pass: []
Fresh: ['Rahul', 'Niranjan', 'Viraj']
Reusable: ['Rahul', 'Niranjan', 'Viraj']

What happened here: The first list(gen) drains the generator and gives you all three names. The second call finds the gate already passed, so it returns an empty list, no warning at all. The fix depends on what you need. If you only need a fresh pass, just call the generator function again to get a brand new generator. If you need to loop more than once, save the values into a real list first, then reuse that list as many times as you like. The trade off is memory: a list keeps everything, a generator does not.

Common Mistakes

❌ Mistake 1: trying to index a generator

gen = (x**2 for x in range(10))
# gen[3]  # TypeError: 'generator' object is not subscriptable
# Generators have no random access, only sequential next()
# Convert to list first if you need indexing: list(gen)[3]

A generator has no index. There is no item number 3 sitting in memory, because the values are produced one at a time and forgotten. Asking for gen[3] raises TypeError: 'generator' object is not subscriptable. If you genuinely need random access, build a list with list(gen) first, then index that.

❌ Mistake 2: forgetting to prime a send()-based generator

def acc():
    total = 0
    while True:
        value = yield total
        total += value

g = acc()
# g.send(10)  # TypeError: can't send non-None value to a just-started generator
next(g)        # Must prime first!
g.send(10)     # Now it works

A brand new generator is parked at the very top, before any yield. There is no waiting yield to receive your value, so calling g.send(10) straight away raises TypeError: can't send non-None value to a just-started generator. Run next(g) once to advance it to the first yield, and then send() works fine.

Conclusion

Python generators are lazy iterators you define with yield instead of return. They hand back values on demand, use almost no memory no matter how big the data gets, and chain together into clean pipelines. For simple cases, a generator expression gives you the same power in one line. And yield from lets one generator delegate straight to another. Remember the big rule too: a generator is single use, so reach for a list when you need more than one pass.

Generators use yield to pause and resume, quietly remembering every local variable in between. Closures pull off a related trick with a twist: they remember variables from an enclosing function even after that function has already returned. In the closures tutorial you will see exactly how that works, and why it is the foundation that makes decorators possible. And if you want to jump around or catch up on earlier topics, browse every post at the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Python Generators

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: Your API service reads a 10 GB log file for a daily report, and memory usage spikes until the process gets killed. What do you check first?

Look for any place the code materializes the whole file at once: f.readlines(), f.read().split("\n"), or a list comprehension over all lines. The fix is to iterate lazily: loop directly over the file object (which yields one line at a time) and chain generator stages for filtering and parsing. That way only one line lives in memory at any moment, so memory stays flat no matter how large the file grows.

Q: A teammate reports that your function returns correct results the first time it is called, but an empty result the second time, with no error. What is the likely bug?

Something is iterating the same generator twice. Generators are single use: once exhausted, every further iteration silently produces nothing. Check whether a generator object is stored in a variable and looped over more than once, or passed to two consumers. Fix it by recreating the generator for each pass, converting it to a list if the data fits in memory, or using itertools.tee() when two consumers genuinely need independent copies.

Q: What happens when you call next() on an exhausted generator, and why do for loops not crash on it?

Calling next() on an exhausted generator raises StopIteration. A for loop catches that exception internally and treats it as the normal end of the loop, which is why iteration just stops cleanly. One related detail worth mentioning in interviews: if a StopIteration is raised inside a generator body, Python converts it to a RuntimeError (PEP 479), so you cannot use it to end a generator early; use return instead.

Q: Can a generator function contain a return statement? What happens to the returned value?

Yes. return inside a generator ends iteration immediately, and the returned value is attached to the resulting exception as StopIteration.value. A plain for loop never sees that value, but yield from does: the expression result = yield from sub_gen() evaluates to the sub-generator’s return value. This is how delegating generators pass final results upward.

Q: What does gen.close() do, and what is GeneratorExit?

gen.close() raises a GeneratorExit exception at the point where the generator is paused. Any try/finally or context manager inside the generator runs its cleanup, which is how generators holding files or connections release them. The generator must then exit: if it catches GeneratorExit and yields another value anyway, Python raises a RuntimeError. Python also calls close() automatically when a generator is garbage collected.

Q: How can you check whether a generator is created, running, suspended, or closed?

Use inspect.getgeneratorstate(gen), which returns one of GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED, or GEN_CLOSED. It is handy when debugging why send() fails: a GEN_CREATED generator has not reached its first yield yet, so it must be primed with next() before it can receive a value.

Try It Yourself

Write a generator function chunk(iterable, size) that takes any iterable and yields lists of size elements at a time. For example, list(chunk(range(10), 3)) should return [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]. Notice the last chunk has only one item, since 10 does not divide evenly by 3, and your function must still yield that leftover. This is a real utility used for database batch inserts and Application Programming Interface (API) pagination, so it is worth getting right.

Go deeper: the official Python documentation covers every edge case of this topic.

Related Posts

Previous: Python: Iterators and the Iterator Protocol

Next: Python: Advanced Comprehensions, Nested, Generator Expressions, Performance

Series Home: Python + AI/ML Tutorial Series