Python - Generators & the yield Keyword in Python

Generators are a special type of function in Python that allow you to produce a sequence of values one at a time, instead of computing and storing all values at once in memory.

1. What is a Generator?

A generator is a function that:

Uses the yield keyword instead of return
Produces values lazily (on demand)
Maintains its state between iterations

Unlike a normal function, which runs completely and returns a single result, a generator pauses execution each time it yields a value and resumes from where it left off.

2. Basic Example

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

Usage:

for num in count_up_to(5):
    print(num)

Output:

Explanation:

The function does not return all values at once
Each call to yield sends back a value and pauses the function
When the loop asks for the next value, execution resumes from the last point

3. How `yield` Works Internally

When a generator function is called:

It does not execute immediately
It returns a generator object

Example:

gen = count_up_to(3)
print(gen)

Output:

<generator object count_up_to at ...>

To get values, you use:

next() function
Or iterate using a loop

Example:

print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3

After all values are exhausted, calling next() raises StopIteration.

4. Difference Between `return` and `yield`

Feature	return	yield
Output	Single value	Multiple values (sequence)
Execution	Ends function	Pauses function
Memory	Stores full result	Generates on demand

5. Why Generators are Important

Memory Efficiency

Consider this:

def get_numbers():
    return [i for i in range(1000000)]

This creates a full list in memory.

Now using a generator:

def get_numbers():
    for i in range(1000000):
        yield i

This:

Does not store all numbers
Produces one number at a time
Saves significant memory

6. Generator Expressions

Similar to list comprehensions, but use parentheses instead of brackets.

List comprehension:

nums = [i*i for i in range(5)]

Generator expression:

nums = (i*i for i in range(5))

Difference:

List stores all values
Generator computes values when needed

7. Real-World Use Cases

Reading Large Files

def read_file(file):
    for line in file:
        yield line

This allows processing file line by line instead of loading entire file.

Infinite Sequences

def infinite_counter():
    i = 0
    while True:
        yield i
        i += 1

This can run indefinitely without consuming large memory.

Data Pipelines

Generators can be chained together:

def square(nums):
    for n in nums:
        yield n * n

def even(nums):
    for n in nums:
        if n % 2 == 0:
            yield n

8. Advantages of Generators

Memory efficient
Faster for large data processing
Cleaner and more readable for pipelines
Useful for streaming data

9. Limitations

Cannot access elements by index
Can only be iterated once
Debugging can be slightly harder

10. Key Takeaway

Generators allow you to handle large or complex data efficiently by:

Producing values only when needed
Avoiding unnecessary memory usage
Enabling lazy evaluation