Iterables & Iterators

We have seen that the methods `__iter__()` and `__next__()` must be implemented for an object to be an iterator object. The implementation of these methods is known as the _iterator protocol_.

If we desire to create our own custom iterator class, we must implement the iterator protocol, meaning we need to have a class that defines at minimum the `__iter__()` and `__next__()` methods.

To look at a scenario where we might require our own custom iterator, imagine we are receiving a shipment of new fish that we can now sell in our pet store. We don't have any classes to manage our fish inventory, so we need to create a custom class to do so. If we wanted to track the available fish inventory, our custom class initializer may look something like this:

```py
class FishInventory:
  def __init__(self, fishList):
      self.available_fish = fishList
```

By default, custom classes are not iterable. We can't just go around plugging our custom classes into `for` loops and expecting any results! This presents a problem if the class we are working with needs the ability to iterate. 

When we create a `FishInventory` class object, we want to iterate over all the fish available within `self.available_fish`. If we attempt to directly iterate over our custom `FishInventory` class object, we will receive an error because we have not yet implemented the iterator protocol for this custom class. To make the `FishInventory` class iterable, we can simply define `__iter__()` and `__next__()` methods.

Defining these methods is discussed in the next exercise, but first, let's see what happens if we attempt to iterate our FishInventory class object directly.


Custom Iterators I

Good job! In this lesson, we covered:
* Iterables and iterators and how they differ.
* Using the `iter()` funtion to create an iterator.
* Using the `next()` function to manually iterate over an iterator.
* How `for` loops use iterables and iterators.
* How to write custom iterators by implementing the `__iter__()` and `__next__()` methods.
* How to use built-in `itertools` including `count()`, `chain()` and `combinations()`.

There is much more to discover about [iterators](https://docs.python.org/3/glossary.html#term-iterator), [iterables](https://docs.python.org/3/glossary.html#term-iterable), and [itertools](https://docs.python.org/3/library/itertools.html?highlight=itertools)! Click the respective links to read more.

Let’s practice these concepts some more!


Review

Let's return to our `for` loop from before: 

```py
for food_brand in dog_foods:
    print(food_brand + " has " + str(dog_foods[food_brand]) + " bags")
```

Under the hood, the first step that the `for` loop has to do is to convert our dictionary (the iterable) of `dog_foods` to an _iterator object_. An iterator object is a special object that represents a stream of data that we can operate on. To accomplish this, it uses a built-in function called [`iter()`](https://docs.python.org/3/library/functions.html#iter): 

```py
dog_food_iterator = iter(dog_foods)
```
We can see the new object by printing it: 
```py
print(dog_food_iterator)
```

This would output our new iterator object:
```bash
<dict_keyiterator object at 0x....> 

# Note: The memory address is omitted since it varies on the system you run the script on. 
```

Here is a visual representation, using the graphic we saw earlier:
![Iter method](https://static-assets.codecademy.com/Courses/Intermediate-Python/Iterables4-final.gif)

To go behind the scenes even further, `iter(dog_foods)` is actually calling a method defined within the iterable called `__iter__()`. All iterables have this `__iter__()` method defined. We can even use the Python built-in function `dir()` to show that our `dog_foods` dictionary (iterable) has a defined method called `__iter__()`.

```py
print(dir(dog_foods))
```

If we examined the output (shortened for brevity), we can spot the `__iter__` property
```
['__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__'...<----- iter method
```

In summary, the `__iter__()` method simply returns the iterator object that allows us to iterate over the iterable. Calling `dog_foods.__iter__()` will retrieve the same iterator object as calling `iter(dog_foods)`. This means that the built-in function `iter()` and the iterable’s method `__iter__()` can be used interchangeably. While the object itself might not seem super useful just yet, we'll see how to manipulate the stream of data inside of it in the next exercises. 

Now that we have taken a peek under the hood, let's practice creating our own iterator objects from iterables using `iter()` and `__iter__()`.


Iterator Objects: __iter__() and iter()

A _combinatoric_ iterator will perform a set of statistical or mathematical operations on an input iterable.

A useful itertool that is a combinatoric iterator is the `combinations()` itertool. This itertool will produce an iterator of tuples that contain combinations of all elements in the input.

```py
combinations(iterable, r)
```

The `combinations()` itertool takes in two inputs, the first is an iterable, and the second is a value `r` that represents the length of each combination tuple.

The return type of `combinations()` is an iterator that can be used in a `for` loop or can be converted into an iterable type using  `list()` or a `set()`.

To show how it’s used, suppose we have a list of even numbers and we want all possible combinations of 2 even numbers:

```py
import itertools
even = [2, 4, 6]
even_combinations = list(itertools.combinations(even, 2))
print(even_combinations)
```
Here we: 

* Import the module `itertools`.
* Create an iterator using `combinations()` with the `list` of even numbers as the first argument and `2` as the second argument.
* Set `even_combinations` equal to a `list` of the elements in the iterator returned from `combinations()`.
* Print `even_combinations`. The resulting list of `2` member tuples are the combinations of all 3 members of `even`:
```bash
[(2, 4), (2, 6), (4, 6)]
```

Now, let's try using a combinations itertool within our pet store!

Combinatoric Iterator: Combinations

Now that we understand how iterators work under the hood, we have all the pieces to put the big picture together. Let's look back at the following `dog_foods` dictionary and the `for` loop that performs the iteration:

```py
dog_foods = {
  "Great Dane Foods": 4,
  "Min Pip Pup Foods": 10,
  "Pawsome Pup Foods": 8
}
for food_brand in dog_foods:
  print (food_brand + " has " + str(dog_foods[food_brand]) + " bags")
```

To summarize, the three main steps are: 

1. The `for` loop will first retrieve an iterator object for the `dog_foods` dictionary using `iter()`.

2. Then, `next()` is called on each iteration of the `for` loop to retrieve the next value. This value is set to the `for` loop's variable, `food_brand`.

3. On each `for` loop iteration, the print statement is executed, until finally, the `for` loop executes a call to `next()` that raises the `StopIteration` exception. The `for` loop then exits and is finished iterating. 

Let's review the process in a visual format, before moving on to learn about creating our very own custom iterators! 

Iterators and For Loops

Now that we used an iterator's `__iter__()` method to create an iterator object, how does our `for` loop know which value to retrieve on each iteration?

Well, the iterator object has a method called `__next__()`, which retrieves the iterator’s next value.  Let's take a look using our SKU iterable for our shop: 

```py
sku_list = [7046538, 8289407, 9056375, 2308597]
sku_iterator = iter(sku_list)
next_sku = sku_iterator.__next__()
print(next_sku)
```

Running this code would produce the following result for `next_sku`:
```bash
7046538
```
  
Similarly to `__iter__()` and `iter()`, there is a Python built-in function called `next()` that we can use in place of calling the `__next__()` method. Calling `next()` simply calls the iterator object's `__next__()` method. Here is the same script but using `next()`:

```py
sku_list = [7046538, 8289407, 9056375, 2308597]
sku_iterator = iter(sku_list)
next_sku = next(sku_iterator)
print(next_sku)
```

Running this code is exactly the same as running the code above using the `__next__()` method and produces the same `next_sku` result:
```bash
7046538
```

But how does the iterator object know when to stop retrieving values? Does it keep calling `__next__()` forever? Well, luckily `__next__()` method will raise an exception called `StopIteration` when all items have been iterated through.

If we call `__next__()` a total of 5 times, one more than the total number of SKUs in our list, we will see the `StopIteration` exception raise on the last `__next__()` call:

```py
sku_list = [7046538, 8289407, 9056375, 2308597]
sku_iterator = iter(sku_list)
for i in range(5):
  next_sku = sku_iterator.__next__()
  print(next_sku)
```

Running this code will produce the following output:
```bash
7046538
8289407
9056375
2308597
```

Followed by: 
```error
Traceback (most recent call last):
  File "main.py", line 24, in <module>
    next_sku = sku_iterator.__next__()
StopIteration
```

In summary, we can finally see why we needed to create the iterator object in the previous exercise. Creating it, allows us to utilize `next()` or `__next__` to work with the stream of data one piece at a time. 

Now, let's practice getting the hang of retrieving individual iterator object values! 

 Iterator Objects:__next__() and next()

While building our own custom iterator classes can be useful, Python offers a convenient, built-in module named _itertools_ that provides the ability to create complex iterator manipulations. These iterator operations can input either a single iterable or a combination of them. 

There are three categories of itertool iterators:
* **Infinite**: Infinite iterators will repeat an infinite number of times. They will not raise a StopIteration exception and will require some type of stop condition to exit from. 
* **Input-Dependent**: Input-dependent iterators are terminated by the input iterable(s) sequence length. This means that the smallest length iterable parameter of an input-dependent iterator will terminate the iterator.
* **Combinatoric**: Combinatoric iterators are iterators that are combinational, where mathematical functions are performed on the input iterable(s).

We can use the itertools module by simply supplying an import statement at the top of the module like this:
```py
import itertools
```

While we will only cover specific itertools for each three of these itertool types, the full list of itertools can be found in the official [Python documentation](https://docs.python.org/3/library/itertools.html).


Python’s Itertools: Built-in Iterators

An infinite iterator will repeat an infinite number of times with no endpoint and no `StopIteration` exception raised. Infinite iterators are useful when we have unbounded streams of data to process.

A useful itertool that is an infinite iterator is the `count()` itertool. This infinite iterator will count from a first value until we provide some type of stop condition. The base syntax of the function looks like this:  

```py
count(start,[step])
```

The first argument of `count()` is the value where we start counting from. The second argument is an optional step that will return `current value + step`. The step value can be positive, negative, and an integer or float number. It will always default to 1 if not provided. 

To show how it’s used in a scenario, suppose we want to quickly count up and print all even numbers from 0 to 20.

We first import our itertools module and then create a loop (this can be a `while` loop or a `for` loop), that will iterate through our `count()` iterator:
```py
import itertools

for i in itertools.count(start=0, step=2):
  print(i)
  if i >= 20:
    break
```

Here is what happens in the script:

* We set our `start` argument to `0` so that we start counting from `0`.

* We set our `step` argument to `2` so that way we increment +2 on each iteration.

* We create a stop condition, which is `i >= 20`, otherwise this `for` loop would continue printing forever!

And our output becomes: 
```bash
0
2
4
6
8
10
12
14
16
18
20
```

Let's use the `count()` itertool to manage our pet store!


Infinite Iterator: Count

To iterate over a custom class we must implement the iterator protocol by defining the `__iter__()` and `__next__()` methods. In most cases the two methods can do the following:

- The `__iter__()` method must always return the iterator object itself. Typically, this is accomplished by returning `self`. It can also include some class member initializing. 

- The `__next__()` method must either return the next value available or raise the `StopIteration` exception. It can also include any number of operations.

Let's return to our custom `FishInventory` class:

```py
class FishInventory:
  def __init__(self, fishList):
      self.available_fish = fishList
```
We want to make the `FishInventory` class iterable to see all the available fish. To make it iterable, we first define the `__iter__()` method. We can initialize a class member within the `__iter__()` method called `index` that will help us track the current position we're in within the `self.available_fish` list.

```py
class FishInventory:
  def __init__(self, fishList):
      self.available_fish = fishList

  def __iter__(self):
    self.index = 0
    return self
```
Notice that the `__iter__()` method returns itself since this class will be an iterator object. The `__iter__()` method can return other iterator objects, but typically the object itself is returned here by using `return self`.

Then, we define the `__next__()` method. Recall that we can perform operations inside this method, like incrementing class members or traversing a `for` loop for instance.

```py
class FishInventory:
  def __init__(self, fishList):
      self.available_fish = fishList

  def __iter__(self):
    self.index = 0
    return self

  def __next__(self):
    fish_status = self.available_fish[self.index] + " is available!"
    self.index += 1
    return fish_status
```
We return the next available fish status within a string value and increment our class member `index` by 1.

Iterating over this class object will eventually error out since we fail to do any checking of our `index` value against the length of the `self.available_fish` list.  We can avoid this and cleanly stop the iterator by raising the `StopIteration` exception in our `__next__()` method. Here, we'll modify our `__next__()` method to raise `StopIteration` if `index` exceeds the length of `available_fish`.

```py
class FishInventory:
  def __init__(self, fishList):
      self.available_fish = fishList

  def __iter__(self):
    self.index = 0
    return self

  def __next__(self):
    if self.index < len(self.available_fish):
      fish_status = self.available_fish[self.index] + " is available!"
      self.index += 1
      return fish_status
    else:
      raise StopIteration
```

Let's practice implementing the iterator protocol in a custom class!

Custom Iterators II

An input-dependent iterator will terminate based on the length of one or more input values. They are great for working with and modifying existing iterators.

A useful itertool that is an input-dependent iterator is the `chain()` itertool. `chain()` takes in one or more iterables and combine them into a single iterator. Here is what the base syntax looks like: 

```py
chain(*iterables)
```

The input value of `chain()` is one or more iterables of the same or varying iterable types. For example, we could use the `chain()` itertool to combine a list and a set into one iterator.

To show how it's used in a scenario, suppose we want to combine a `list` containing odd numbers and a `set` containing even numbers:

```py
import itertools

odd = [5, 7, 9]
even = {6, 8, 10}

all_numbers = itertools.chain(odd, even)

for number in all_numbers:
  print(number)
```
The above example:
* Imports the `itertools` module.
* Sets `all_numbers` to the iterator returned by the itertool `chain()`.
* Uses the `list` iterable `odd` and the `set` iterable `even` as the arguments to `chain()`. 
* Implements a `for` loop using the iterator in `all_numbers`
* Prints the results, which will be:
```bash
5
7
9
8
10
6
```

The output is finite since the input iterables, `odd` and `even` are also finite. Note that Python sets are not ordered so the last 3 numbers in this example's output will not always be in the initialized order. 

Let's use `chain()` to work with SKU iterables in our pet store!

Input-Dependent Iterator: Chain

In Python, an _iterable_ is an object that is capable of being looped through one element at a time. We commonly use iterables to perform the process of iteration and it is the backbone for how we perform consistent operations on sets of data.

While these may be new terminologies, we have actually been using iterables frequently.  Dictionaries, lists, tuples, and sets are all classified as iterables!  We have even performed the process of iteration anytime we used looping mechanisms such as  `for` loops. In this lesson, we are going to dive deeper into how looping (iteration) works under the hood and the role iterables play in it!  

Before we dive into the details, let's imagine we are the new owners of a pet store and need to keep track of certain inventory such as bags of food and their quantities. We can accomplish this using a dictionary.

Suppose we have the following information we need to store:

Dog Food Brand | Quantity
--- | ---
Great Dane Foods| 4 bags
Min Pip Pup Foods| 10 bags
Pawsome Foods| 8 bags

<br>  

Using a dictionary, our data would take this form:

```py
dog_foods = {
    "Great Dane Foods": 4,
    "Min Pin Pup Foods": 10,
    "Pawsome Pups Foods": 8
}
```

Now if we wanted to display all the dog brands and their respective quantities, we could use a common loop such as the `for` loop to accomplish this goal.   Our program might look something like this:

```py
for food_brand in dog_foods:
    print(food_brand + " has " + str(dog_foods[food_brand]) + " bags")
```

In this case, our iterable is the dictionary called `dog_foods` and the `for` loop performs the process of iteration that allows us to access each element. It looks a bit like magic, but what is actually happening under the hood here? How does the `for` loop iterate over each dictionary member and know when to stop iterating once the end is reached?

In the next sections, we will dive into answering these questions. For now, observe a high-level overview of the inner workings of our `dog_foods` loop.