Tutorials Python Python Iterators & Generators
Ch 43 / 58
Python Intermediate 14 min read Chapter 43 of 58

Python Iterators & Generators

Every Python for loop secretly uses the iterator protocol — calling __iter__ and __next__ behind the scenes. Generators take this further with the yield keyword: they let you write iteration logic as ordinary functions that pause and resume, producing values lazily one at a time without storing the entire sequence in memory. This makes them perfect for processing large datasets, building pipelines, and creating infinite sequences — three of the most powerful patterns in production Python code.

What you'll learn in this chapter
Iterables, iterators and the iterator protocol
Building custom iterators with __iter__ & __next__
Generator functions with yield — lazy evaluation
Generator expressions — compact generator syntax
Infinite sequences and data pipelines
yield from — delegating to sub-generators
Advertisement

The Iterator Protocol — What Powers for Loops

Python's for loop is syntactic sugar for the iterator protocol — a two-method interface. Any object that implements these methods is an iterator and can be used in a for loop, list(), next(), and anywhere Python expects iterable data.

Iterator protocol — two required methods
1 __iter__(self) called once — setup
Returns the iterator object itself. Called at the start of a for loop or when iter(obj) is called. An object with only __iter__ is an iterable.
def __iter__(self): return self
2 __next__(self) called repeatedly — produce values
Returns the next value. Called by each iteration of the loop or by next(iterator). When exhausted, raise StopIteration.
def __next__(self): raise StopIteration ← signals end
3 StopIteration signals exhaustion
The for loop catches StopIteration and stops cleanly. You never need to catch it yourself — Python handles it automatically.
raise StopIteration ← for loop exits gracefully
Python — iterator_protocol.py
# What a for loop actually does under the hood:
# for x in [1, 2, 3]:
#     print(x)
# is equivalent to:

_iter = iter([1, 2, 3])          # calls list.__iter__()
while True:
    try:
        x = next(_iter)              # calls list.__next__()
        print(x)
    except StopIteration:
        break

# Iterable vs Iterator distinction:
my_list = [1, 2, 3]             # ITERABLE — has __iter__
my_iter = iter(my_list)          # ITERATOR — has __iter__ AND __next__

print(next(my_iter))             # 1
print(next(my_iter))             # 2
print(next(my_iter))             # 3
# next(my_iter)                   → StopIteration

# A list is iterable but NOT an iterator
print(hasattr(my_list, "__iter__"))   # True — iterable
print(hasattr(my_list, "__next__"))   # False — not an iterator!
print(hasattr(my_iter, "__next__"))   # True — iterator

# Anything that works in a for loop is iterable:
# list, tuple, str, dict, set, range, file, generator...

Custom Iterators — Implementing __iter__ & __next__

By implementing the iterator protocol in your own class, you can make any object iterable — countdown timers, database cursors, file chunk readers, number sequences, anything. The class must track its own state to know where it is between calls.

Python — custom_iterator.py
# ── Countdown iterator ────────────────────────────────
class Countdown:
    """Counts down from start to 1."""

    def __init__(self, start):
        self.current = start

    def __iter__(self):     # makes it an iterable
        return self             # return self (also the iterator)

    def __next__(self):
        if self.current < 1:
            raise StopIteration  # signal exhaustion
        value = self.current
        self.current -= 1
        return value

for n in Countdown(5):
    print(n, end=" ")         # 5 4 3 2 1

print(list(Countdown(3)))    # [3, 2, 1]

# ── Separate iterable and iterator ────────────────────
# Better design: iterable creates a fresh iterator each time
class NumberRange:           # ITERABLE
    def __init__(self, start, stop):
        self.start = start
        self.stop  = stop

    def __iter__(self):
        return NumberRangeIterator(self.start, self.stop)

class NumberRangeIterator:   # ITERATOR
    def __init__(self, current, stop):
        self.current = current
        self.stop    = stop

    def __iter__(self): return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        val = self.current
        self.current += 1
        return val

rng = NumberRange(1, 4)
for n in rng: print(n)       # 1 2 3
for n in rng: print(n)       # 1 2 3  ← fresh iterator each time!

Generator Functions — yield Makes it Magic

A generator function looks like a normal function but uses yield instead of return. When called, it doesn't execute the body — instead it returns a generator object (an iterator). The body runs lazily, one yield at a time, pausing after each value.

Generator execution flow — pause and resume
gen = my_generator() ← body does NOT run yet
▼
next(gen) called
Body runs until first yield
▼
yield value → returned to caller
▼
function state preserved here
⏸ Paused — local variables kept in memory
▼
next(gen) called again
Resumes from where it paused → next yield
▼
when no more yields
StopIteration raised automatically
Python — generator_functions.py
# ── Basic generator function ──────────────────────────
def countdown(n):
    print("Starting...")
    while n > 0:
        yield n               # pause here, return n to caller
        n -= 1               # resumes here on next() call
    print("Done!")

gen = countdown(3)          # body does NOT run yet
print(type(gen))              # <class 'generator'>

print(next(gen))              # Starting...  \n  3
print(next(gen))              # 2
print(next(gen))              # 1
# next(gen)                   → Done!  then StopIteration

# ── Generator as for loop ─────────────────────────────
for n in countdown(5):
    print(n, end=" ")         # 5 4 3 2 1

# ── Generator vs class iterator — same result ─────────
def squares(n):                # generator — clean and simple
    for i in range(n):
        yield i * i

print(list(squares(5)))      # [0, 1, 4, 9, 16]

# ── Generator with multiple yields ────────────────────
def alphabet():
    yield "alpha"
    yield "beta"
    yield "gamma"

for letter in alphabet():
    print(letter)              # alpha  beta  gamma

# ── Generator is lazy — only computes what's needed ──
def big_range():
    i = 0
    while True:               # would run forever...
        yield i
        i += 1

gen = big_range()
print([next(gen) for _ in range(5)])  # [0, 1, 2, 3, 4]  ← safe!
Advertisement
K2infocom Python Masterclass & Backend Development
Master Python, Django & Backend APIs with AI
Go from absolute beginner to building production-ready REST APIs, web applications, and ML pipelines. Real projects, live mentoring in Vadodara, and placement support included.
2,400+
Students Enrolled
4.9 ⭐
Google Rating
6 Months
Full Program
100%
Placement Support
Python 3 Django & DRF FastAPI PostgreSQL Machine Learning Docker & AWS Real Projects AI Tools

yield in Depth — send(), throw() and close()

Generators are two-way communication channels. Beyond just producing values, you can send values back into a generator, throw exceptions into it, and close it. This turns generators into powerful coroutine-like objects used in async frameworks.

Python — yield_in_depth.py
# ── yield as an expression — send() ──────────────────
def accumulator():
    """Receive numbers, yield running total."""
    total = 0
    while True:
        value = yield total    # yield total OUT; receive value IN
        if value is None: break
        total += value

gen = accumulator()
next(gen)               # prime the generator (advance to first yield)
print(gen.send(10))     # 10  — total after adding 10
print(gen.send(5))      # 15  — total after adding 5
print(gen.send(20))     # 35  — total after adding 20

# ── generator.return value ────────────────────────────
def gen_with_return():
    yield 1
    yield 2
    return "finished"       # return value stored in StopIteration

g = gen_with_return()
print(next(g))            # 1
print(next(g))            # 2
try:
    next(g)
except StopIteration as e:
    print(e.value)          # finished  ← return value!

# ── throw() — inject exception into generator ─────────
def safe_gen():
    try:
        yield "running"
    except ValueError as e:
        yield f"caught: {e}"

g = safe_gen()
print(next(g))                         # running
print(g.throw(ValueError, "oops"))    # caught: oops

# ── close() — terminate generator ────────────────────
g = countdown(10)
print(next(g))                         # 10
g.close()                               # throws GeneratorExit into generator

Generator Expressions — Compact & Memory Efficient

Generator expressions have the same syntax as list comprehensions but use () instead of []. They produce a generator object — lazy, memory-efficient, and ideal for one-pass processing.

List Comprehension — [x for x in ...]
Builds the entire list in memory immediately. Good when you need random access, multiple passes, or know the data is small.
Generator Expression — (x for x in ...)
Produces values lazily one at a time. Uses O(1) memory regardless of input size. Good for large files, streams, pipelines — when you only need one pass.
Python — generator_expressions.py
import sys

# ── List vs Generator — memory comparison ────────────
lst = [i**2 for i in range(1000)]  # list comprehension  → stores all 1000 values
gen = (i**2 for i in range(1000))  # generator expression → stores nothing

print(sys.getsizeof(lst))           # ~8856 bytes
print(sys.getsizeof(gen))           # 200 bytes (just the generator object!)

# ── Use in functions ──────────────────────────────────
print(sum(i**2 for i in range(10)))  # 285  — no list built!
print(max(i**2 for i in range(10)))  # 81
print(any(i > 5 for i in [1, 3, 7]))  # True  — short-circuits!
print(all(i > 0 for i in [1, 2, 3]))  # True

# ── Filter with condition ─────────────────────────────
evens = (x for x in range(20) if x % 2 == 0)
print(list(evens))               # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

# ── Chaining generator expressions (pipeline) ─────────
numbers  = (x for x in range(100))
filtered = (x for x in numbers if x % 3 == 0)
squared  = (x**2 for x in filtered)
print(list(squared)[:5])         # [0, 9, 36, 81, 144]

# ── Process large file line-by-line ───────────────────
def count_lines_with(filepath, keyword):
    with open(filepath, encoding="utf-8") as f:
        matching = (line for line in f if keyword in line)
        return sum(1 for _ in matching)
# Memory: O(1) — reads one line at a time regardless of file size

Infinite Sequences & Data Pipelines

Generators can produce an infinite stream of values since they compute lazily — no memory is needed for unread values. Use itertools or write your own infinite generators, then take only what you need with next(), slicing, or itertools.islice().

Python — infinite_and_pipelines.py
from itertools import islice, count, cycle, repeat

# ── Infinite counter ──────────────────────────────────
def integers(start=0):
    n = start
    while True:
        yield n
        n += 1

# Take only first 5
print(list(islice(integers(), 5)))  # [0, 1, 2, 3, 4]

# ── Fibonacci generator ───────────────────────────────
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

print(list(islice(fibonacci(), 10)))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

# ── itertools built-in infinite generators ────────────
print(list(islice(count(10, 2), 5)))  # [10, 12, 14, 16, 18]
print(list(islice(cycle("ABC"), 7)))    # ['A','B','C','A','B','C','A']
print(list(islice(repeat(42), 4)))     # [42, 42, 42, 42]

# ── Generator pipeline — process large CSV in memory ─
def read_lines(path):
    with open(path, encoding="utf-8") as f:
        yield from f

def parse_csv(lines):
    for line in lines:
        yield line.strip().split(",")

def filter_active(rows):
    for row in rows:
        if row[-1] == "active":
            yield row

# Pipeline — O(1) memory even for gigabyte files!
pipeline = filter_active(parse_csv(read_lines("users.csv")))

yield from — Delegating to Sub-Generators

yield from iterable is a clean way to yield all values from another iterable or generator — without a manual for loop. It also propagates send(), throw(), and close() to the sub-generator, making it essential for generator composition.

Python — yield_from.py
# ── yield from vs manual loop ─────────────────────────
def flatten_manual(nested):
    for inner in nested:
        for item in inner:
            yield item                # manual delegation

def flatten(nested):
    for inner in nested:
        yield from inner             # cleaner with yield from

data = [[1, 2], [3, 4], [5, 6]]
print(list(flatten(data)))        # [1, 2, 3, 4, 5, 6]

# ── yield from any iterable ───────────────────────────
def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain([1,2], "ab", (3,4))))  # [1, 2, 'a', 'b', 3, 4]

# ── Recursive tree traversal ──────────────────────────
def deep_flatten(item):
    """Flatten arbitrarily nested lists."""
    if isinstance(item, list):
        for subitem in item:
            yield from deep_flatten(subitem)  # recursive!
    else:
        yield item

nested = [1, [2, [3, 4]], [5, [6, [7]]]]
print(list(deep_flatten(nested)))  # [1, 2, 3, 4, 5, 6, 7]

# ── yield from captures return value ─────────────────
def sub_gen():
    yield 1; yield 2
    return "sub done"

def main_gen():
    result = yield from sub_gen()  # captures return value!
    print(f"Sub returned: {result}")
    yield 3

print(list(main_gen()))         # Sub returned: sub done  → [1, 2, 3]

Practice Python Online — Free Compilers

Don't want to install anything yet? Practice all iterator and generator examples from this chapter directly in your browser with these free Python environments.

▶
Online Python 3 Compiler
Write, run and debug Python code in your browser. No installation required. Test every iterator and generator example instantly.
Open Compiler
Replit
Full IDE online
Google Colab
Free GPU notebooks
JupyterLite
Notebooks in browser
Programiz
Beginner-friendly
Chapter Summary
  • An iterable has __iter__ (returns an iterator). An iterator has both __iter__ and __next__. Raising StopIteration signals exhaustion — the for loop catches this automatically.
  • Custom iterators need state stored as instance attributes to know where they are between __next__ calls. Separate iterable and iterator classes allow multiple independent iterations.
  • A generator function contains yield — calling it returns a generator object without executing the body. The body runs lazily, one yield at a time, pausing and preserving all local state.
  • Generator expressions (x for x in ...) are compact, lazy, and memory-efficient. Use them instead of list comprehensions when you only need one pass or the data is large.
  • Generators support send(value) (inject a value), throw(exc) (inject an exception), and close() (terminate). Generator return values are accessible via StopIteration.value.
  • Infinite generators are safe because of lazy evaluation. Use itertools.islice(), next(), or an explicit break to consume only what you need.
  • yield from iterable delegates to a sub-generator or any iterable, replacing manual for-yield loops. It also propagates send()/throw() and captures the sub-generator's return value.
  • Generator pipelines chain multiple generators — filter(parse(read(file))) — processing large data in O(1) memory, one element at a time.
Free Resource
Download our Python Iterators & Generators Cheat Sheet
Iterator protocol, custom iterators, yield, generator expressions, yield from, infinite sequences & pipelines — one printable page.
Get Sheet
Advertisement
Quick Check — Test Your Understanding
Question & Answer
✓ Answer

An iterable is any object with a __iter__ method that returns an iterator. Lists, tuples, strings, and dicts are iterables — you can loop over them. But they are NOT iterators themselves.

An iterator is an object with BOTH __iter__ (returns self) AND __next__ (produces the next value). It maintains state between calls and raises StopIteration when exhausted.

Relationship: every iterator is an iterable, but not every iterable is an iterator. Calling iter(my_list) creates an iterator from a list.

✓ Answer

A function becomes a generator function simply by containing at least one yield statement anywhere in its body.

When you call a generator function, the body does NOT execute. Instead, Python returns a generator object (which is an iterator). The body only runs when you call next() on the generator — it runs until the next yield, then pauses and preserves all local state until the next call.

✓ Answer

A list comprehension [x**2 for x in range(1000000)] builds and stores all 1,000,000 values in memory immediately — using megabytes of RAM.

A generator expression (x**2 for x in range(1000000)) creates a generator object that produces values lazily on demand. It uses only a few hundred bytes regardless of how many values it will produce — O(1) memory. This makes generator expressions essential when processing large files, streams, or API responses.

✓ Answer

yield from iterable delegates iteration to another iterable or generator, producing all its values. It's equivalent to:

for item in iterable:
    yield item

But yield from is better because: it's more readable, it transparently propagates send(), throw(), and close() to the sub-generator, and it captures the sub-generator's return value (accessible as the expression result).

✓ Answer

An infinite generator is safe because values are produced lazily — only computed when requested. The key is to consume a limited number of values:

  • next(gen) — get one value at a time
  • itertools.islice(gen, n) — take exactly n values
  • break in a for loop — stop when condition is met

Never pass an infinite generator to list(), sum(), or any function that tries to consume all values — it will loop forever (or until memory runs out).

MCQ 1 of 3
What exception does a Python iterator raise to signal it is exhausted?
MCQ 2 of 3
What is the memory-efficient alternative to [x**2 for x in range(1000)]?
MCQ 3 of 3
When does the body of a generator function first execute?

Rate Us on Google

4.8 / 5
Enjoying this tutorial?
Your 5-star review helps other learners discover our free tutorials and encourages us to keep creating quality content.
80+ reviews
Write a Google Review
Join WhatsApp Channel