Python Iterators & Generators
Every Python for loop secretly uses the
iterator protocol — calling __iter__ and __next__ behind the scenes. Generators take this further with
the yield keyword: they let you write iteration logic as ordinary
functions that pause and resume, producing values lazily one at a time without storing the entire
sequence in memory. This makes them perfect for processing large datasets, building pipelines, and creating
infinite sequences — three of the most powerful patterns in production Python code.
The Iterator Protocol — What Powers for Loops
Python's for loop is syntactic sugar for the
iterator protocol — a two-method interface. Any object that implements these methods is an
iterator and can be used in a for loop, list(), next(), and anywhere Python
expects iterable data.
for loop or when iter(obj) is
called. An object with only __iter__ is an iterable.next(iterator). When exhausted, raise StopIteration.StopIteration and stops
cleanly. You never need to catch it yourself — Python handles it automatically.# What a for loop actually does under the hood:
# for x in [1, 2, 3]:
# print(x)
# is equivalent to:
_iter = iter([1, 2, 3]) # calls list.__iter__()
while True:
try:
x = next(_iter) # calls list.__next__()
print(x)
except StopIteration:
break
# Iterable vs Iterator distinction:
my_list = [1, 2, 3] # ITERABLE — has __iter__
my_iter = iter(my_list) # ITERATOR — has __iter__ AND __next__
print(next(my_iter)) # 1
print(next(my_iter)) # 2
print(next(my_iter)) # 3
# next(my_iter) → StopIteration
# A list is iterable but NOT an iterator
print(hasattr(my_list, "__iter__")) # True — iterable
print(hasattr(my_list, "__next__")) # False — not an iterator!
print(hasattr(my_iter, "__next__")) # True — iterator
# Anything that works in a for loop is iterable:
# list, tuple, str, dict, set, range, file, generator...
Custom Iterators — Implementing __iter__ & __next__
By implementing the iterator protocol in your own class, you can make any object iterable — countdown timers, database cursors, file chunk readers, number sequences, anything. The class must track its own state to know where it is between calls.
# ── Countdown iterator ────────────────────────────────
class Countdown:
"""Counts down from start to 1."""
def __init__(self, start):
self.current = start
def __iter__(self): # makes it an iterable
return self # return self (also the iterator)
def __next__(self):
if self.current < 1:
raise StopIteration # signal exhaustion
value = self.current
self.current -= 1
return value
for n in Countdown(5):
print(n, end=" ") # 5 4 3 2 1
print(list(Countdown(3))) # [3, 2, 1]
# ── Separate iterable and iterator ────────────────────
# Better design: iterable creates a fresh iterator each time
class NumberRange: # ITERABLE
def __init__(self, start, stop):
self.start = start
self.stop = stop
def __iter__(self):
return NumberRangeIterator(self.start, self.stop)
class NumberRangeIterator: # ITERATOR
def __init__(self, current, stop):
self.current = current
self.stop = stop
def __iter__(self): return self
def __next__(self):
if self.current >= self.stop:
raise StopIteration
val = self.current
self.current += 1
return val
rng = NumberRange(1, 4)
for n in rng: print(n) # 1 2 3
for n in rng: print(n) # 1 2 3 ← fresh iterator each time!
Generator Functions — yield Makes it Magic
A generator function looks like a normal function but uses yield instead of return. When called, it
doesn't execute the body — instead it returns a generator object (an iterator). The body runs
lazily, one yield at a time, pausing after each value.
# ── Basic generator function ──────────────────────────
def countdown(n):
print("Starting...")
while n > 0:
yield n # pause here, return n to caller
n -= 1 # resumes here on next() call
print("Done!")
gen = countdown(3) # body does NOT run yet
print(type(gen)) # <class 'generator'>
print(next(gen)) # Starting... \n 3
print(next(gen)) # 2
print(next(gen)) # 1
# next(gen) → Done! then StopIteration
# ── Generator as for loop ─────────────────────────────
for n in countdown(5):
print(n, end=" ") # 5 4 3 2 1
# ── Generator vs class iterator — same result ─────────
def squares(n): # generator — clean and simple
for i in range(n):
yield i * i
print(list(squares(5))) # [0, 1, 4, 9, 16]
# ── Generator with multiple yields ────────────────────
def alphabet():
yield "alpha"
yield "beta"
yield "gamma"
for letter in alphabet():
print(letter) # alpha beta gamma
# ── Generator is lazy — only computes what's needed ──
def big_range():
i = 0
while True: # would run forever...
yield i
i += 1
gen = big_range()
print([next(gen) for _ in range(5)]) # [0, 1, 2, 3, 4] ← safe!
yield in Depth — send(), throw() and close()
Generators are two-way communication channels. Beyond just producing values, you can send values back into a generator, throw exceptions into it, and close it. This turns generators into powerful coroutine-like objects used in async frameworks.
# ── yield as an expression — send() ──────────────────
def accumulator():
"""Receive numbers, yield running total."""
total = 0
while True:
value = yield total # yield total OUT; receive value IN
if value is None: break
total += value
gen = accumulator()
next(gen) # prime the generator (advance to first yield)
print(gen.send(10)) # 10 — total after adding 10
print(gen.send(5)) # 15 — total after adding 5
print(gen.send(20)) # 35 — total after adding 20
# ── generator.return value ────────────────────────────
def gen_with_return():
yield 1
yield 2
return "finished" # return value stored in StopIteration
g = gen_with_return()
print(next(g)) # 1
print(next(g)) # 2
try:
next(g)
except StopIteration as e:
print(e.value) # finished ← return value!
# ── throw() — inject exception into generator ─────────
def safe_gen():
try:
yield "running"
except ValueError as e:
yield f"caught: {e}"
g = safe_gen()
print(next(g)) # running
print(g.throw(ValueError, "oops")) # caught: oops
# ── close() — terminate generator ────────────────────
g = countdown(10)
print(next(g)) # 10
g.close() # throws GeneratorExit into generator
Generator Expressions — Compact & Memory Efficient
Generator expressions have the same syntax as list comprehensions but use () instead of []. They produce a generator
object — lazy, memory-efficient, and ideal for one-pass processing.
import sys
# ── List vs Generator — memory comparison ────────────
lst = [i**2 for i in range(1000)] # list comprehension → stores all 1000 values
gen = (i**2 for i in range(1000)) # generator expression → stores nothing
print(sys.getsizeof(lst)) # ~8856 bytes
print(sys.getsizeof(gen)) # 200 bytes (just the generator object!)
# ── Use in functions ──────────────────────────────────
print(sum(i**2 for i in range(10))) # 285 — no list built!
print(max(i**2 for i in range(10))) # 81
print(any(i > 5 for i in [1, 3, 7])) # True — short-circuits!
print(all(i > 0 for i in [1, 2, 3])) # True
# ── Filter with condition ─────────────────────────────
evens = (x for x in range(20) if x % 2 == 0)
print(list(evens)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# ── Chaining generator expressions (pipeline) ─────────
numbers = (x for x in range(100))
filtered = (x for x in numbers if x % 3 == 0)
squared = (x**2 for x in filtered)
print(list(squared)[:5]) # [0, 9, 36, 81, 144]
# ── Process large file line-by-line ───────────────────
def count_lines_with(filepath, keyword):
with open(filepath, encoding="utf-8") as f:
matching = (line for line in f if keyword in line)
return sum(1 for _ in matching)
# Memory: O(1) — reads one line at a time regardless of file size
Infinite Sequences & Data Pipelines
Generators can produce an infinite stream of values since they compute lazily — no
memory is needed for unread values. Use itertools or write your own
infinite generators, then take only what you need with next(), slicing, or
itertools.islice().
from itertools import islice, count, cycle, repeat
# ── Infinite counter ──────────────────────────────────
def integers(start=0):
n = start
while True:
yield n
n += 1
# Take only first 5
print(list(islice(integers(), 5))) # [0, 1, 2, 3, 4]
# ── Fibonacci generator ───────────────────────────────
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 10)))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
# ── itertools built-in infinite generators ────────────
print(list(islice(count(10, 2), 5))) # [10, 12, 14, 16, 18]
print(list(islice(cycle("ABC"), 7))) # ['A','B','C','A','B','C','A']
print(list(islice(repeat(42), 4))) # [42, 42, 42, 42]
# ── Generator pipeline — process large CSV in memory ─
def read_lines(path):
with open(path, encoding="utf-8") as f:
yield from f
def parse_csv(lines):
for line in lines:
yield line.strip().split(",")
def filter_active(rows):
for row in rows:
if row[-1] == "active":
yield row
# Pipeline — O(1) memory even for gigabyte files!
pipeline = filter_active(parse_csv(read_lines("users.csv")))
yield from — Delegating to Sub-Generators
yield from iterable is a clean way to yield all
values from another iterable or generator — without a manual for loop. It
also propagates send(), throw(), and
close() to the sub-generator, making it essential for generator
composition.
# ── yield from vs manual loop ─────────────────────────
def flatten_manual(nested):
for inner in nested:
for item in inner:
yield item # manual delegation
def flatten(nested):
for inner in nested:
yield from inner # cleaner with yield from
data = [[1, 2], [3, 4], [5, 6]]
print(list(flatten(data))) # [1, 2, 3, 4, 5, 6]
# ── yield from any iterable ───────────────────────────
def chain(*iterables):
for it in iterables:
yield from it
print(list(chain([1,2], "ab", (3,4)))) # [1, 2, 'a', 'b', 3, 4]
# ── Recursive tree traversal ──────────────────────────
def deep_flatten(item):
"""Flatten arbitrarily nested lists."""
if isinstance(item, list):
for subitem in item:
yield from deep_flatten(subitem) # recursive!
else:
yield item
nested = [1, [2, [3, 4]], [5, [6, [7]]]]
print(list(deep_flatten(nested))) # [1, 2, 3, 4, 5, 6, 7]
# ── yield from captures return value ─────────────────
def sub_gen():
yield 1; yield 2
return "sub done"
def main_gen():
result = yield from sub_gen() # captures return value!
print(f"Sub returned: {result}")
yield 3
print(list(main_gen())) # Sub returned: sub done → [1, 2, 3]
Practice Python Online — Free Compilers
Don't want to install anything yet? Practice all iterator and generator examples from this chapter directly in your browser with these free Python environments.
- An iterable has
__iter__(returns an iterator). An iterator has both__iter__and__next__. RaisingStopIterationsignals exhaustion — theforloop catches this automatically. - Custom iterators need state stored as instance attributes to know where they are between
__next__calls. Separate iterable and iterator classes allow multiple independent iterations. - A generator function contains
yield— calling it returns a generator object without executing the body. The body runs lazily, oneyieldat a time, pausing and preserving all local state. - Generator expressions
(x for x in ...)are compact, lazy, and memory-efficient. Use them instead of list comprehensions when you only need one pass or the data is large. - Generators support
send(value)(inject a value),throw(exc)(inject an exception), andclose()(terminate). Generator return values are accessible viaStopIteration.value. - Infinite generators are safe because of lazy evaluation. Use
itertools.islice(),next(), or an explicitbreakto consume only what you need. yield from iterabledelegates to a sub-generator or any iterable, replacing manual for-yield loops. It also propagatessend()/throw()and captures the sub-generator's return value.- Generator pipelines chain multiple generators —
filter(parse(read(file)))— processing large data in O(1) memory, one element at a time.
An iterable is any object with a __iter__ method
that returns an iterator. Lists, tuples, strings, and dicts are iterables — you can loop over them.
But they are NOT iterators themselves.
An iterator is an object with BOTH __iter__ (returns self) AND __next__ (produces the next value). It maintains state between calls
and raises StopIteration when exhausted.
Relationship: every iterator is an iterable, but not every iterable is an
iterator. Calling iter(my_list) creates an iterator from a list.
A function becomes a generator function simply by containing at least one yield statement anywhere in its body.
When you call a generator function, the body does NOT execute. Instead,
Python returns a generator object (which is an iterator). The body only runs when you
call next() on the generator — it runs until the next yield, then pauses and preserves all local state until the next call.
A list comprehension [x**2 for x in range(1000000)] builds and
stores all 1,000,000 values in memory immediately — using megabytes of RAM.
A generator expression (x**2 for x in range(1000000)) creates a generator object that
produces values lazily on demand. It uses only a few hundred bytes regardless of how many values it
will produce — O(1) memory. This makes generator expressions essential when processing large files,
streams, or API responses.
yield from iterable delegates iteration to another iterable or
generator, producing all its values. It's equivalent to:
for item in iterable:
yield item
But yield from is better because: it's
more readable, it transparently propagates send(), throw(), and close() to the
sub-generator, and it captures the sub-generator's return value (accessible as the expression result).
An infinite generator is safe because values are produced lazily — only computed when requested. The key is to consume a limited number of values:
next(gen)— get one value at a timeitertools.islice(gen, n)— take exactly n valuesbreakin aforloop — stop when condition is met
Never pass an infinite generator to list(), sum(), or any function
that tries to consume all values — it will loop forever (or until memory runs out).
[x**2 for x in range(1000)]?
Rate Us on Google
Join Our Learning Community
Connect with thousands of learners, get daily tips, ask questions and stay updated with new tutorials.