Every collection item in python is iterable

An iterable provides iterator for loop and comprehensions i.e list comprehensions, set comprehensions, dictionary comprehensions, and generator expressions. Don't fret if you don't understand these terms, we'll explain them in future lessons.

One of the standout attributes of Horace Learning we use close-to-real-scenarios for our examples.

Lets use a converstaion between a teacher and a student to explain iterables.

Teacher: "Today, we're going to talk about iterables and generators in Python. Can anyone give me an example of an iterable?"

Student: "Um, is a list an iterable?"

Teacher: "Excellent! Yes, a list is an iterable. Can you think of why we might use iterables?"

Student: "I think we use them in for loops, right?"

Teacher: "Correct! Iterables are very useful in for loops. Now, let's talk about generators. They're a special kind of iterable that generates values on-the-fly."

Student: "On-the-fly? What does that mean?"

Teacher: "It means the values are created as you need them, rather than all at once. This can be very memory-efficient for large datasets."

Student: "Oh, I see. Can you show us an example?"

Teacher: "Of course! Let's create a simple generator that yields the squares of numbers."

One easy way to demystify computer science is to think in your native language not the abstract machine language.

• For our first exercise, we will:
  • Get all the words from the teacher's first sentence.
  • Count the number of words in the sentence.
  • Finally, find repeated words in the sentence.

Get the Teacher's First Sentence​

sentence = "Today, we're going to talk about iterables and generators in Python."

Get all the words from the teacher's first sentence.​

words = sentence.split()
print("All words:", words)

Count the number of words in the sentence.​

word_count = len(words)
print("Word count:", word_count)

Find repeated words in the sentence.​

word_frequency = {}
repeated_words = []

for word in words: # Convert to lowercase to ignore case
word = word.lower() # Remove punctuation
word = word.strip('.,')

    # Count word frequency
    if word in word_frequency:
        word_frequency[word] += 1
        if word_frequency[word] == 2:
            repeated_words.append(word)
    else:
        word_frequency[word] = 1

print("Repeated words:", repeated_words)

NB: There are more efficient ways to find repeated words such asusing a set to store the words and then use a dictionary to count the frequency of each word, or using wordcloud with matplotlib. However, this lesson teaches iteration with the for keyword

Exercise 1:

• Print the top 10 most frequent words in the conversation.

Solution​

import string

# Combine all the dialogue into one string
conversation = """
Today, we're going to talk about iterables and generators in Python. Can anyone give me an example of an iterable?
Um, is a list an iterable?
Excellent! Yes, a list is an iterable. Can you think of why we might use iterables?
I think we use them in for loops, right?
Correct! Iterables are very useful in for loops. Now, let's talk about generators. They're a special kind of iterable that generates values on-the-fly.
On-the-fly? What does that mean?
It means the values are created as you need them, rather than all at once. This can be very memory-efficient for large datasets.
Oh, I see. Can you show us an example?
Of course! Let's create a simple generator that yields the squares of numbers.
"""

# Clean and split the text
words = conversation.lower().translate(str.maketrans('', '', string.punctuation)).split()

# Count word frequencies
word_freq = {}
for word in words:
    if len(word) > 3:  # Ignore short words
        word_freq[word] = word_freq.get(word, 0) + 1

# Sort words by frequency
sorted_words = sorted(word_freq.items(), key=lambda x: x[1], reverse=True)

# Display top 10 words
print("Top 10 most frequent words:")
for word, freq in sorted_words[:10]:
    print(f"{word}: {'*' * freq}")

How Does Iteration Work?​

• Python checks if the object is iterable by calling __iter__() method. i.e iter(object)
• If the object is iterable, Python calls __iter__() method to get an iterator.
• If it is not iterable, Python raises a TypeError exception:

  TypeError: 'int' object is not iterable

• If __iter__() method is not implementd, but __getitem__() is, Python uses the __getitem__() method to iterate over the object by index starting from 0.
• The iterator is an object with a __next__() method.
• The __next__() method returns the next item in the sequence.
• If the iterator is exhausted, Python raises a StopIteration exception.

Handson 2:

Understanding Iterables and Iterators in Python

1. Python Checks if the Object is Iterable​

class SimpleIterable:
    def __iter__(self):
        return iter([1, 2, 3])

simple = SimpleIterable()
iterator = iter(simple)  # This calls __iter__()
print(list(iterator))  # Output: [1, 2, 3]

If the object is not iterable, Python raises a TypeError exception:​

try:
    iter(42)
except TypeError as e:
    print(e) # Output: 'int' object is not iterable

If __iter__() method is not implemented, but __getitem__() is, Python uses the __getitem__() method to iterate over the object by index starting from 0.​

class SimpleIterable:
    def __getitem__(self, index):
        return [1, 2, 3][index]

simple = SimpleIterable()
iterator = iter(simple)
print(next(iterator))  # Output: 1

Iterator with __next__() method​

class SimpleIterable:
    def __iter__(self):
        return iter([1, 2, 3])

simple = SimpleIterable()
iterator = iter(simple)
print(next(iterator))  # Output: 1

In summary, a pythonic object is iterable if it has __iter__() or __getitem__() method.

Clean code guide

• If you will be iterating over an object, its not necessary to check if the object is iterable. Python will raise a TypeError exception if the object is not iterable. No point re-inventing the wheel. The built in iter() function will mostly be used by python itself than by the developer.
• Use try except to catch the TypeError exception instead of doing explicit checks. we will discuss exceptions in future lessons

Further Exercises​

• Count all sentences by the teacher.
• Count all sentences by the student.
• Count all words by the teacher.
• Count all words by the student.
• Count all punctuation marks by the teacher.
• Count all punctuation marks by the student.
• Count all vowels by the teacher.
• Count all vowels by the student.
• Count all consonants by the teacher.
• Count all consonants by the student.
• Count all numbers by the teacher.
• Count all numbers by the student.
• Count all special characters by the teacher.
• Count all special characters by the student.

Generators are a type of iterable in Python that can be used to generate a sequence of values. They are defined using a function that contains one or more yield expressions. When a generator function is called, it returns a generator object that can be iterated over lazily, yielding one value at a time.

Here's an example of a generator function that generates a sequence of even numbers:

def even_numbers(n: int) -> int:
    """
    Generate a sequence of even numbers up to n.

    Args:
        n (int): The upper limit of the sequence (exclusive).

    Yields:
        int: The next even number in the sequence.
    """
    for i in range(n):
        if i % 2 == 0:
            yield i

The yield keyword is used to return a value from the generator function and pause the execution of the function. The generator object can then be used to continue the execution of the function from the point where it was paused.

To get a value from the generator, you can use the next() function.

# Create a generator object that yields even numbers up to 10
even_nums = even_numbers(10)

# Get and print the first even number (0)
print(next(even_nums))

# Get and print the second even number (2)
print(next(even_nums))

# Get and print the third even number (4)
print(next(even_nums))

You can also use a for loop to iterate over the generator object.

def print_even_numbers(limit: int) -> None:
    """
    Print even numbers up to the given limit.

    Args:
        limit (int): The upper limit for generating even numbers (exclusive).

    Returns:
        None
    """
    for num in even_numbers(limit):
        print(num)

print_even_numbers(10)

You can also convert a generator object to a list. (But don't do this)

even_nums = even_numbers(10)
print(list(even_nums))

Note: Please don't convert a generator object to a list if you only need to iterate over it once. This will defeat the purpose of using a generator.

Note: The generator object is not a list, it is an iterator. You can only iterate over it once. If you need to use the values multiple times, you should convert the generator object to a list.

The Generator always remembers its state between calls and you can resume from there with no issues or state errors.

Benefits of Generator​

1. Memory Efficiency: Generators give values on-the-fly, so they don't require as much memory as traditional loops. This makes them ideal for processing large datasets.
2. Performance: Generators can be faster than traditional loops because they don't require as much memory.
3. Pause and Resume: Generators can pause and resume execution, which makes them ideal for processing large datasets.

Note: A generator doesn't raise a StopIteration exception, it is a feature of iterators and you don't have to worry about it. It just exits to make it clear that the generator has reached the end of the sequence.

Generator Expression​

Generator expressions are similar to list comprehensions, but they return a generator object instead of a list.

Note: Remember, in python, if its iterable, then there is a comprehension. e.g List Comprehension, Set Comprehension, Dictionary Comprehension, and now Generator Comprehension.

# Create a generator expression that yields even numbers from 0 to 9
even_nums = (i for i in range(10) if i % 2 == 0)

# Print the generator object (not the actual values)
print(even_nums)

Here is a comparison of a generator expression and a list comprehension.

# Create a generator expression for even numbers from 0 to 9
even_nums = (i for i in range(10) if i % 2 == 0)

# Create a list comprehension for even numbers from 0 to 9
even_nums_list = [i for i in range(10) if i % 2 == 0]

# Print the generator object (not the actual values)
print(even_nums)

# Print the list of even numbers
print(even_nums_list)

Notice generator expression is wrapped in parentheses (), while list comprehension is wrapped in square brackets [].

Use Cases for Generator Expressions​

User generator expressions when you need to generate a sequence of values on-the-fly, but don't need to store all the values in memory. Consider the cost of storing intermediate values in memory when choosing between a generator expression and a list comprehension. Note: When writting any comprehension, if the code spans more than one line, you may be better off with traditional for loop. For the following reasons:

• Readability
• Debugging
• Complexity
• PEP8
• Flake8 However, consider the following exceptions to the rule:
• Line lenght limit rules
• Complex operations
• Team code guide/rule
• Generator expressions are always more efficient for large datasets than a loop that build a list.

Examples​

# List comprehension to create a list of squares
squares = [x**2 for x in range(10)]

# Explanation:
# - This creates a list called 'squares'
# - It uses a list comprehension to generate squares of numbers
# - 'x**2' calculates the square of x
# - 'for x in range(10)' iterates over numbers 0 to 9
# - The result is a list of squares: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Multi-line comprehension (might be better as a loop)​

# Define a complex calculation using list comprehension
complex_calculation = [
    (x**2 + y**2) / (x + y)  # Calculate (x^2 + y^2) / (x + y)
    for x in range(1, 10)    # Outer loop: x from 1 to 9
    for y in range(1, 10)    # Inner loop: y from 1 to 9
    if (x + y) % 2 == 0      # Only include results where x + y is even
]

# Result:
# [1.0, 2.5, 2.0, 3.5, 3.0, 4.5, 4.0, 5.5, 5.0, 2.5, 4.0, 3.5, 5.0, 4.5, 6.0, 5.5, 7.0, 2.0, 3.5, 5.0, 4.5, 6.0, 5.5, 7.0, 6.5]

The above might be clearer as:​

# Initialize an empty list to store the results
complex_calculation = []

# Outer loop: iterate over x from 1 to 9
for x in range(1, 10):
    # Inner loop: iterate over y from 1 to 9
    for y in range(1, 10):
        # Check if the sum of x and y is even
        if (x + y) % 2 == 0:
            # Calculate the result using the formula (x^2 + y^2) / (x + y)
            result = (x**2 + y**2) / (x + y)
            # Append the calculated result to the complex_calculation list
            complex_calculation.append(result)

# At this point, complex_calculation contains all the results

Thank you for reading this article. I hope you found it helpful. Next lesson, we will discuss map, filter vs list comprehension.

Please follow, like and subscribe to my channel and social media pages.