Python Generators

Generators are a type of iterable in Python that can be used to generate a sequence of values. They are defined using a function that contains one or more yield expressions. When a generator function is called, it returns a generator object that can be iterated over lazily, yielding one value at a time.

Here's an example of a generator function that generates a sequence of even numbers:

def even_numbers(n: int) -> int:
    """
    Generate a sequence of even numbers up to n.

    Args:
        n (int): The upper limit of the sequence (exclusive).

    Yields:
        int: The next even number in the sequence.
    """
    for i in range(n):
        if i % 2 == 0:
            yield i

The yield keyword is used to return a value from the generator function and pause the execution of the function. The generator object can then be used to continue the execution of the function from the point where it was paused.

Traditional Loop	Generator Yield
Computes all values at once	Generates values on-demand
Stores all values in memory	Stores only the current value
Uses more memory for large datasets	Memory-efficient for large datasets
Faster for small datasets	Slightly slower due to overhead
Cannot pause execution	Can pause and resume execution
Good for finite, small sequences	Excellent for large or infinite sequences

To get a value from the generator, you can use the next() function.

# Create a generator object that yields even numbers up to 10
even_nums = even_numbers(10)

# Get and print the first even number (0)
print(next(even_nums))

# Get and print the second even number (2)
print(next(even_nums))

# Get and print the third even number (4)
print(next(even_nums))

You can also use a for loop to iterate over the generator object.

def print_even_numbers(limit: int) -> None:
    """
    Print even numbers up to the given limit.

    Args:
        limit (int): The upper limit for generating even numbers (exclusive).

    Returns:
        None
    """
    for num in even_numbers(limit):
        print(num)

print_even_numbers(10)

You can also convert a generator object to a list. (But don't do this)

even_nums = even_numbers(10)
print(list(even_nums))

Note: Please don't convert a generator object to a list if you only need to iterate over it once. This will defeat the purpose of using a generator.

Note: The generator object is not a list, it is an iterator. You can only iterate over it once. If you need to use the values multiple times, you should convert the generator object to a list.

The Generator always remembers its state between calls and you can resume from there with no issues or state errors.

Benefits of Generator

1. Memory Efficiency: Generators give values on-the-fly, so they don't require as much memory as traditional loops. This makes them ideal for processing large datasets.
2. Performance: Generators can be faster than traditional loops because they don't require as much memory.
3. Pause and Resume: Generators can pause and resume execution, which makes them ideal for processing large datasets.

Note: A generator doesn't raise a StopIteration exception, it is a feature of iterators and you don't have to worry about it. It just exits to make it clear that the generator has reached the end of the sequence.

Generator Expression

Generator expressions are similar to list comprehensions, but they return a generator object instead of a list.

Note: Remember, in python, if its iterable, then there is a comprehension. e.g List Comprehension, Set Comprehension, Dictionary Comprehension, and now Generator Comprehension.

# Create a generator expression that yields even numbers from 0 to 9
even_nums = (i for i in range(10) if i % 2 == 0)

# Print the generator object (not the actual values)
print(even_nums)

Here is a comparison of a generator expression and a list comprehension.

# Create a generator expression for even numbers from 0 to 9
even_nums = (i for i in range(10) if i % 2 == 0)

# Create a list comprehension for even numbers from 0 to 9
even_nums_list = [i for i in range(10) if i % 2 == 0]

# Print the generator object (not the actual values)
print(even_nums)

# Print the list of even numbers
print(even_nums_list)

Notice generator expression is wrapped in parentheses (), while list comprehension is wrapped in square brackets [].

Use Cases for Generator Expressions

User generator expressions when you need to generate a sequence of values on-the-fly, but don't need to store all the values in memory. Consider the cost of storing intermediate values in memory when choosing between a generator expression and a list comprehension. Note: When writting any comprehension, if the code spans more than one line, you may be better off with traditional for loop. For the following reasons:

• Readability
• Debugging
• Complexity
• PEP8
• Flake8 However, consider the following exceptions to the rule:
• Line lenght limit rules
• Complex operations
• Team code guide/rule
• Generator expressions are always more efficient for large datasets than a loop that build a list.

Examples

# List comprehension to create a list of squares
squares = [x**2 for x in range(10)]

# Explanation:
# - This creates a list called 'squares'
# - It uses a list comprehension to generate squares of numbers
# - 'x**2' calculates the square of x
# - 'for x in range(10)' iterates over numbers 0 to 9
# - The result is a list of squares: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Multi-line comprehension (might be better as a loop)

# Define a complex calculation using list comprehension
complex_calculation = [
    (x**2 + y**2) / (x + y)  # Calculate (x^2 + y^2) / (x + y)
    for x in range(1, 10)    # Outer loop: x from 1 to 9
    for y in range(1, 10)    # Inner loop: y from 1 to 9
    if (x + y) % 2 == 0      # Only include results where x + y is even
]

# Result:
# [1.0, 2.5, 2.0, 3.5, 3.0, 4.5, 4.0, 5.5, 5.0, 2.5, 4.0, 3.5, 5.0, 4.5, 6.0, 5.5, 7.0, 2.0, 3.5, 5.0, 4.5, 6.0, 5.5, 7.0, 6.5]

The above might be clearer as:

# Initialize an empty list to store the results
complex_calculation = []

# Outer loop: iterate over x from 1 to 9
for x in range(1, 10):
    # Inner loop: iterate over y from 1 to 9
    for y in range(1, 10):
        # Check if the sum of x and y is even
        if (x + y) % 2 == 0:
            # Calculate the result using the formula (x^2 + y^2) / (x + y)
            result = (x**2 + y**2) / (x + y)
            # Append the calculated result to the complex_calculation list
            complex_calculation.append(result)

# At this point, complex_calculation contains all the results

Thank you for reading this article. I hope you found it helpful. Next lesson, we will discuss map, filter vs list comprehension.

Please follow, like and subscribe to my channel and social media pages.