Generators

2 min read 5 runnable examples Quiz included

A generator is the easy way to make an iterator. It's a function that uses yield instead of return.

Python

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for x in countdown(5):
    print(x, end=" ")
print("Liftoff!")

Compare that with the 15-line Countdown class from the previous lesson: same behaviour, a quarter of the code.

How yield works

Calling a generator function doesn't run it. It returns a generator object. Each next() runs the function until the next yield, hands out that value, and pauses, keeping all local variables, until the next request:

Python

def steps():
    print("  starting")
    yield 1
    print("  resumed after 1")
    yield 2
    print("  resumed after 2")
    yield 3

gen = steps()
print("created")
print(next(gen))
print(next(gen))
print(next(gen))

Generators are lazy

Values are produced only when asked for, so a generator can describe huge, even infinite, sequences without using memory:

Python

def fibonacci():
    a, b = 0, 1
    while True:            # never ends, and that's fine
        yield a
        a, b = b, a + b

for i, f in enumerate(fibonacci()):
    if i == 12:
        break
    print(f, end=" ")

Generator expressions

Like a list comprehension, but in round brackets. It produces items lazily instead of building a list:

Python

squares_list = [n * n for n in range(1_000_000)]    # builds a million-item list
squares_gen = (n * n for n in range(1_000_000))     # builds nothing yet

import sys
print(sys.getsizeof(squares_list), "bytes")
print(sys.getsizeof(squares_gen), "bytes")
print(sum(n * n for n in range(1_000_000)))         # brackets can be dropped inside a call

A pipeline of generators

Generators chain together into pipelines, each step processing one item at a time:

Python

lines = ["  ERROR disk full", "INFO started", "  error timeout ", "INFO ok", "ERROR crashed"]

def clean(rows):
    for row in rows:
        yield row.strip()

def only_errors(rows):
    for row in rows:
        if row.lower().startswith("error"):
            yield row

for problem in only_errors(clean(lines)):
    print(problem)

With a real log file of millions of lines, this uses almost no memory, because only one line is in flight at a time.

Finished this lesson?Mark it complete to earn +20 XP and move your progress on.