Learn common functional programming skills.

Functional Programming

We may find ourselves in situations where we are working with JSON Data. The JSON format is the preferred way to structure data such that it may be transmitted over the internet. An API call usually responds with data formatted in JSON. We can process JSON data using the higher-order functions we have been working with. 

However, unlike a CSV file, we cannot use an iterator when we read data from a JSON file. The file's entire contents are read and stored in a Python dictionary. If a JSON object contains another JSON object, the dictionary will contain another dictionary. Unfortunately, the `json` library in Python does not provide a method to produce an iterator that we can use to read data similar to a CSV; we will have to work with the `dict` type.

To illustrate this in practice, we will process data from the sample JSON file: `cities.json`, which contains the following data:
<details>
<summary>Click to see JSON data</summary>

```json
{
   "city": [
       {
            "name": "New York",
            "country": "United States of America",
            "coordinates": [40.7128, -74.0060],
            "continent": "North America"
        },
        {
            "name":"Los Angeles",
            "country": "United States of America",
            "coordinates": [34.0522, 118.2437],
            "continent": "North America"
        },
        {
            "name": "Montreal",
            "country": "Canada",
            "coordinates": [45.5017, -73.5673],
            "continent": "North America"
        },
        {
            "name": "Toronto",
            "country": "Canada",
            "coordinates": [43.6532, -79.3832],
            "continent": "North America"
        }
     ]
}
```

</details>

<br>


To work with the JSON file, we must first be open it and read it as we did for a CSV file:

```py
import json
with open('cities.json') as json_file:
  cities = json.load(json_file)
```

In the JSON file, there is an object called `city` that consists of an array of JSON objects containing data relevant to a particular city. When the JSON file is read, it will produce a dictionary containing one entry: a map between the string "city" and a list of dictionaries that represent data relevant to each contained city. The dictionary structure will be:

```py
{"city":[{"name": "name_of_city", "country": "country_of_city", "coordinates":[latitude, longitude], "continent":"continent_of_city"}...]}.
```
The data formatted this way is hard to read and susceptible to errors when using it. It would be better to use `map()` to store this data as a `namedtuple`.

```py
import json
from collections import namedtuple

city = namedtuple("city", ["name", "country", "coordinates", "continent"])

with open('cities.json') as json_file:
  data = json.load(json_file) 
  # cities is a dictionary with one entry: {"cities":[{"name": name, "country":country, "coordinates":[lat, long], "continent":continent}...]

# Extract the collection of cities from the dictionary - this is a list of dictionaries.
cities_list = data["cities"]

cities = map(lambda x: city(x["name"], x["country"], x["coordinates"], x["continent"]), cities_list)
```
The `cities` iterable can now store the entries of the JSON file in a much more readable way.

The other higher-order functions can be used in the same way as `map()` with `cities` now being the supplied iterable.


Processing Data From a JSON File

In this exercise, we will explore how to use `reduce()` and `filter()` together. Let's consider the following example. We have a tuple representing the menu items for a fictitious restaurant: The Codecademy Steakhouse. We wish to find the least expensive appetizer on the menu. We can express this process by the sentence: "filter the menu for appetizers and return the least expensive one." Let's do this imperatively first.

The starting code:
<details>
<summary>Click to see starting code</summary>

```py
from collections import namedtuple
from functools import reduce

# Prices are in USD
menu_item = namedtuple("menu_item", ["name", "dish_type", "price"])

jsp = menu_item("Jumbo Shrimp Platter", "Appetizer", 29.95)
lc = menu_item("Lobster Cake", "Appetizer", 30.95)
scb = menu_item("Sizzling Canadian Bacon", "Appetizer", 9.95)
ccc = menu_item("Codecademy Crab Cake", "Appetizer", 32.95)
cs = menu_item("Caeser Salad", "Salad", 14.95)
mgs = menu_item("Mixed Green Salad", "Salad", 11.95)
cp = menu_item("Codecademy Potatoes", "Side", 34.95)
mp = menu_item("Mashed Potatoes", "Side", 14.95)
rs = menu_item("Ribeye Steak", "Entree", 75.95)
phs = menu_item("Porter House Steak", "Entree", 131.95)

menu = (jsp, lc, scb, ccc, cs, mgs, cp, mp, rs, phs)
```
</details>
<br>
Imperative solution:

```py
# apps only contains items with dish_type == "Appetizer"
apps = [i for i in menu if i.dish_type == "Appetizer"] 
cheapest_app = apps[0] 
for i in apps:
  if i.price < cheapest_app.price:
    cheapest_app = i

print(cheapest_app) # Output will be: menu_item("Sizzling Canadian Bacon", "Appetizer", 9.95)
```

We can eliminate the `for` loop and do this declaratively by using `filter()` and `reduce()` like this:

```py
cheapest_app = reduce(lambda x, y: x if x.price < y.price else y, filter(lambda x: x.dish_type == "Appetizer", menu))

print(cheapest_app) # Output will be: menu_item("Sizzling Canadian Bacon", "Appetizer", 9.95)
```

In the `reduce()` function, the lambda `lambda x, y: x if x.price < y.price else y` returns the cheaper of the two `menu_items` compared by their price.

The `filter()` function returns an iterable with all `menu_items` that have "Appetizer" as `dish_type`. This iterable is then passed into `reduce()` to be processed by its lambda function. Combining these two functions converts the imperative code into a one-line solution!


Reducing a Filtered Collection

In this exercise, we will see how we can process the data that is stored in a CSV file.

When working with a file that contains a large amount of data, generating every possible record using `tuple()` is inefficient. Recall from the previous exercise that `map()` returns an iterator that we can use to bring in data. We can apply the higher-order functions to this iterator to process the data and only bring in the relevant set of records. For example, let's say we only want to retain tuples for houses built after 1985. We can do this by applying `filter()` with the lambda `lambda q: q[2] > 1985` like so:

```py
h = filter(lambda q: q[2] > 1985, map(lambda x: house(int(x[0]), int(x[1]), int(x[2]), int(x[3])), reader))

houses = tuple(h)
print(houses) # Print out tuples of house built after 1985
```  
Now let's say we want the most expensive house built after 1985. We can use `reduce()` to do this for us like so:

```py
h = filter(lambda q: q[2] > 1985, map(lambda x: house(int(x[0]), int(x[1]), int(x[2]), int(x[3])), reader))
houses = tuple(h)

from functools import reduce

most_expensive = reduce(lambda x, y: x if x.list_price > y.list_price else y, houses)
print(most_expensive) # Print out tuples of house built after 1985
``` 

When working with CSV files, it is important to ensure they are properly formatted; some entries might have leading or trailing whitespaces that we must remove, for example.

Processing Data From a CSV File

In this exercise, we will review the `map()`, `filter()`, and `reduce()`, higher-order functions provided by Python. These functions accept an iterable and a processing function as arguments and return another iterable. Similar to the exercise on lambdas, this section should serve as a refresher. You may recognize these functions from intermediate Python content. If you need more than a quick refresher and feel stuck at all during this exercise, we recommend you review this [article](https://www.codecademy.com/courses/learn-intermediate-python-3/articles/built-in-higher-order-functions-in-python) as an aid.

The first function we will look at is `filter()`, which converts the following imperative code:

```py
nums = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

# filter_values is not a higher-order function
def filter_values(predicate, lst):

  # Mutable list required because this example is imperative, not declarative
  ret = []
  for i in lst:
    if predicate(i):
      ret.append(i)
  return ret

filtered_numbers = filter_values(lambda x: x % 2 == 0, nums) 

print(filtered_numbers) 

# This will output the list: [2, 4, 6, 8, 10]
```

into the following declarative code:


```py
nums = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

filtered_numbers = filter(lambda x: x % 2 == 0, nums) 

print(tuple(filtered_numbers))

# This will output the tuple: (2, 4, 6, 8, 10)
```

`map()` converts the following imperative code:

```py
nums = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

def mapper(function, lst):
  ret = []
  for i in lst:
    ret.append(function(i))
  return ret

mapped_numbers  = mapper(lambda x: x*x, nums)

print(tuple(mapped_numbers))

# This will output: (1, 4, 9, 16, 25, 36, 49, 64, 81, 100)
```
Into the following declarative code:

```py
numbers = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

mapped_numbers = map(lambda x: x*x, numbers) 

print(tuple(mapped_numbers))

# This will also output: (1, 4, 9, 16, 25, 36, 49, 64, 81, 100)
```

The `reduce()` function accepts an iterable and a two-parameter function (no more than two) known as an *"accumulator"*. It uses the accumulator to recursively process the contents of the iterable and "reduce" them to one value. An example application of `reduce()` would be to compute the sum of all numbers in a list.

The sum of all elements in a tuple written imperatively:

```py
nums = (2, 6, 7, 9, 1, 4, 8)

sum = 0

for i in nums:
  sum += i

print(sum) # Output: 37
```

Using `reduce()` to find the sum declaratively: 

```py
"""
In Python 3, the `reduce()` function has been moved to the `functools` library, so we need to import it before we can use it.
"""
from functools import reduce

nums = (2, 6, 7, 9, 1, 4, 8)

reduced_nums = reduce(lambda x, y: x + y, nums) # reduced_nums is a number

print(reduced_nums) # Output: 37
```
The lambda in `reduce()` requires exactly two parameters.

Higher-order functions are powerful tools that you can use to significantly shorten code, thereby leading to cleaner and more concise solutions. In later exercises, we will explore how to combine these functions. 


Review of filter(), map(), and reduce()

Lambda functions are crucial in functional programming as they allow the production of neat and concise functions that require other functions as an argument. You should be familiar with lambda functions from previous courses on Python, but if you are not (or would like a refresher), this exercise will serve as a brief review.

A lambda function is a short [anonymous function](https://www.codecademy.com/resources/docs/python/functions/anonymous-functions?page_ref=catalog) that can accept several parameters but only returns one value. Lambdas can be stored as a variable or defined inline in the accepting function.

Consider the following example:

We determine the cost of a two-dimensional sheet of metal by multiplying the price per square-meter times the area of the sheet. We can cut the sheet into either a rectangle or a triangle. To model this, let's write a general function in Python that calculates the price of a sheet of metal. The function will accept the following parameters: 
* price per square-meter
* a tuple of dimensions, 
* a function to compute area. 
The function will return the price of a sheet of metal.

Recall from geometry:
- Area of a triangle: 0.5 * (base * height)
- Area of a rectangle: base * height

Writing the function without lambdas, we have:

```py
def rect(b, h):
  return b * h

def tri(b, h):
  return 0.5 * (b * h)

# ppsm: price per square meter
# dim: dimensions tuple
def total_cost(ppsm, dim, area):
  return ppsm * area(dim[0], dim[1])

print(total_cost(3, (5, 5), rect)) # Rectangular sheet costing 75 units
print(total_cost(4, (6, 7), tri)) # Triangular sheet costing 84 units
```

To make the code shorter, we can replace the `rect(b,h)` and the `tri(b,h)` functions with lambdas like so:

 ```py
# ppsm: price per square meter
# dim: dimensions tuple
def total_cost(ppsm, dim, area):
  return ppsm * area(dim[0], dim[1])

print(total_cost(3, (5, 5), lambda b, h: b*h)) # Rectangular sheet costing 75 units
print(total_cost(4, (6, 7), lambda b, h: 0. 5 * b*h)) # Triangular costing 84 units

```
Lambda functions can be stored in a variable like so:

```py
rect = lambda x, y: x * y
tri = lambda x, y: 0.5 * x * y
```

Lambdas are a very powerful tool in Python. However, they have a drawback in that they are only suited for "short and fast" functions. Long, complicated functions cannot be written as lambdas.

Review of Lambda Functions

The fundemantal concepts behind functional programming: first class functions, no side effects, lazy evaluations, recursion, statelessness, immutable data, pure functions.

Get ready to become a master of functional programming! This content branches off some content you may have run into from our Intermediate Python. We will do a quick review of:
* tuples
* lambda functions
* `map()`, `reduce()`, `filter()`
* working with CSV and JSON files

Then we will dive into more advanced problems using these tools!

Introduction

In this exercise, we will be focussing on the benefits of using `reduce()` and `map()` together. 

Consider the example of having a dictionary representing a cost of an item sale called `costs` which maps an item name to a tuple containing the total number of units sold and the price per unit. A dictionary entry would look like this: 

```python
"name": (total_number_of_item_sold, price_per_item)
``` 

We wish to find the total cost of all items sold. If this were a list or a tuple, we could simply apply `reduce()` on it to find the sum. This, however, would present a problem if we attempt this with a dictionary. 

The lambda provided to `reduce()` requires that the two parameters and the returned value be of the same type. For example, in the lambda `lambda x, y: x*y`, the `x`, `y`, and return type are all integers. As you can see, you cannot directly reduce a dictionary to a number because they are not of the same type; we must process the data in the dictionary first. 

We can use `map()` to iterate through the dictionary and compute the cost of every item sold. We can potentially store this in a tuple and then reduce that tuple to a single number, the total cost.

Note: when passing a dictionary as an iterable, the function will iterate through the list of the dictionaries keys.

Let's look at an example with the following program:

```py
from functools import reduce

# Dictionary entry: {"name: (number_or_units_sold, price_per_unit_GBP)}
costs = {"shirt": (4, 13.00), "shoes":(2, 80.00), "pants":(3, 100.00), "socks":(5, 5.00)}

k = reduce(lambda x, y: x+y, map(lambda q: costs[q][0] * costs[q][1], costs))
print(k) # Output will be a total cost of: 537.0 GBP

```

This dictionary is passed into `map()` along with the lambda `lambda q: costs[q][0] * costs[q][1]`. The lambda function takes the price tuple and generates a total_cost_per_item by multiplying the `number_of_units_sold` (`costs[q][0]`) by the `price_per_unit_GBP` (`costs[q][1]`). The lambda in the `reduce()` function is now working strictly with integers to sum them up and returns a total cost of £537.

We could have done this exercise using `namedtuple`, but we excluded it for brevity. Using `map()` to process a dictionary is key when working with the JSON format, as we will see in a later exercise.

 Reducing a Mapped Collection

Functional programming is widely applicable in the data science domain as higher-order functions can be used to process data files efficiently. One of the most common formats for a data file is a CSV file (comma-separated value). In this exercise and the next, we will work with the three higher-order functions to process data contained in a CSV file. For this exercise, we will use a CSV file containing housing data as an example.

The file **zillow.csv** contains housing data collected by the American real-estate company Zillow for the city of Tallahassee Florida. In this exercise we will use `map()` to import the data from the CSV file and represent it using a `namedtuple`.

The data in the file is represented by the following table (which shows the first five lines):

| Index | Square footage | Year | List price (USD) |
| -------- | -------------------- | ------- | -------------------- | 
| 1 | 2222 | 1981 | 250000 |
| 2 | 1628 | 2009 | 185000 |
| 3 | 3824 | 1954 | 399000 |
| 4 | 1137 | 1993 | 150000 |
| 5 | 3560 | 1973 |315000 |

To work with this file, we must first open it and create a *reader* object (an iterator) that will read the file:

```py
import csv

with open('zillow.csv', newline = '') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='|')
```

A CSV file often contains millions of lines of data. Importing the entire contents of a CSV file is impractical as this would occupy too much RAM resulting in poor program performance. to avoid importing all the data at once, `reader` is an iterator object that maintains a pointer to the file and iterates through the data when `next(reader)` is called.

We create a `namedtuple` to represent each record:

```py
from collections import namedtuple

house = namedtuple("house", ["index", "square_footage", "year", "list_price"])
```
We use the `map()` function to read in a line of data and store it in a tuple. Incoming data arrives as a list of strings regardless of their intended type; therefore we must cast each element to convert it to the proper type. The order of the data is dictated by the first line in a CSV file. 

This is done like so:

```py
# Read record into namedtuple: house(index, square_footage, year, list_price)
h = map(lambda x: house(int(x[0]), int(x[1]), int(x[2]), int(x[3])), reader)
```
Because the amount of data in the `zillow` file is small, it would be okay to create a tuple called `houses` and populate it with all individual house tuples generated like so:

```py
print(tuple(h))
```
In the next exercise, we will see how we can use the higher-order functions to process data contained in the CSV file.


Importing Data From a CSV File

Congratulations on completing this module on functional programming in Python. We've covered many topics on how to write neat and efficient solutions using functional programming. Let's briefly recap everything covered so far:

* Functional programming follows the declarative approach to programming; the programmer describes what needs to be done as opposed to how it is done.
* Functional programming relies heavily on immutable data structures.
* It is preferable to use a tuple instead of a list when writing code because a tuple is immutable.
* Using a `namedtuple` to store data leads to a more readable data structure.
* The `namedtuple` collection resides in the `collections` library.
* A `namedtuple` is created like so:
```python
name = ("name", ["property1", "property2", …, "property"])
```
* Representing an object by a `namedtuple` is done like so:
```python
obj = name(property1, property2, …, property)
```
* Referencing an element of a `namedtuple` is done like this:
```python
property = name.property
```
* A lambda function is an anonymous function.
* A lambda function can be stored in a variable like any other data type.
* A lambda function can be defined directly while passing it as an argument of another function.
* The three higher-order functions Python provides are:
  * `map()`
  * `filter()`
  * `reduce()`
* These functions can be supplied as the iterable parameter to any of the higher-order functions. Ex:
  * h = map(lambda, filter(lambda, map(lambda, iterable)))
* Combining `map()` and `filter()` filters an iterable before a mapping function is applied.
* Combining `reduce()` and `filter()` filters an iterable before it is reduced.
* A CSV file can be processed by supplying the CSV reader as the iterable to the higher-order functions.
* A JSON file can only be represented in Python by a dictionary.
* An iterable is not available to read data stored in a JSON file.
* A dictionary must be provided to the higher-order functions when processing a JSON file.

Happy Coding!


Review

Now that you've learned how to combine any two functions, let's see how (and why) we can combine all three! A reason for doing this would be when you need to "filter" a collection before you "map" it (or "map" then "filter") and then "reduce"  it to a single number. To illustrate this in practice, let's revisit the inventory problem from earlier, but this time we are only interested in the total sum of prices of items that sold for less than a certain value.

In this Python exercise, we are given the record of items represented by a dictionary. We are interested in the total sum of prices of items that sold for less than £150. We do this by:
* First "map" the items their individual total cost (_(number of units sold) * (price per unit)_).
* Then eliminate ("filter" out) all items that cost more than £150.
* Then "reduce" the individual costs to a single number that represents the total cost of the items. 

We can do this with the following code:

```py
from functools import reduce

# Dictionary entry: {"name: (number_or_units_sold, price_per_unit_GBP)}

costs = {"shirt": (4, 13.00), "shoes":(2, 80.00), "pants":(3, 100.00), "socks":(5, 5.00), "ties":(3, 14.00), "watch":(1, 145.00)}

k = reduce(lambda x, y: x + y, filter(lambda r: r <= 150.00, map(lambda q: costs[q][0] * costs[q][1], costs)))

print(k) # Output will be a total cost of: 264.0 GBP
```

The focus of the code is:

```py
k = reduce(lambda x, y: x + y, filter(lambda r: r <= 150.00, map(lambda q: costs[q][0] * costs[q][1], costs)))
```

This first computes the total cost of each item using `map()`:

```py
map(lambda q: costs[q][0]*costs[q][1], costs)
```

Then `filter()` is used to eliminate all items that cost more than £150:

```py
filter(lambda r: r <= 150.00, map(lambda q: costs[q][0] * costs[q][1], costs))
```

Then `reduce()` is used to compute the total sum:

```py
reduce(lambda x, y: x + y, filter(lambda r: r <= 150.00, map(lambda q: costs[q][0] * costs[q][1], costs)))
```

This example demonstrates how to use the three functions together.


Combining all Three Higher-Order Functions

Immutable data types are important to use in functional programming as they offer advantages, such as: 
* thread-safe data manipulation
* preventing programmers from accidentally changing a value 

We can create a tuple of tuples with a mix of datatypes, like the following:

```py
student  = (("Scott", 28, 'A'), ("Nicole", 26, 'B'), ("John", 29, 'D')) # mixed elements 
```

This is a great way to store our mixed data. However, there is room for improvement. The student tuple contains records of students in a CS class where each tuple stores the student’s name, age, and the grade they received. Defining a tuple in this manner is prone to errors as it requires the programmer to remember the position of each piece of data in the tuple.

Instead, we can use a `namedtuple` data type from the `collections` library like so:

```py
from collections import namedtuple

# Create a class called student
student = namedtuple("student", ["name", "age", "grade"]) 

# Create tuples for the three students

scott = student("Scott", 28, 'A')
nicole = student("Nicole", 26, 'B')
john = student("John", 29, 'D')
```

We access the student information:

```py
from collections import namedtuple

# create a class called student
student = namedtuple("student", ["name", "age", "grade"]) 

# Create tuples for the three students
scott = student("Scott", 28, 'A')
nicole = student("Nicole", 26, 'B')
john = student("John", 29, 'D')

# Access Scott’s information for example
print(scott.name) # Output: Scott
print(scott.age) # Output: 28
print(scott.grade) # Output: ‘A’
```

Take note that the name of the tuple and the variable that stores the tuple must be identical.

We can package three student tuples neatly into a tuple called `students` like so:

```py
students = (scott, nicole, john)
```
The programmer is no longer required to remember the position of each piece of data as they can reference it using the property name.


Immutable Data types

In this exercise, we will see how we can combine the `map()` and `filter()` functions. 

Conceptually, if you're working with a collection of items and find yourself saying, "I need to map only values that have property *x*," you will likely need to use `map()` and `filter()` together. in English, "I need to map filtered values" translates into Python like this: 

> map(mapping_function, filter(predicate, iterable))

Let's take a look at an example. 

We have a tuple representing the records of students in various math classes. The tuple will be structured in the following way: `student(name, grade, course)`. As an example, `student("Peter", 'B', 101)` will represent a student named Peter who received a grade of B in a Math 101 course. 

We want to register students who receive a grade of B or better in their respective math courses for a special advanced math course: Math 201. To create the record for the course, we will filter all students with grade B or higher and map their math course to 201. 

The initial grade in the new tuple will contain the letter 'X' to represent that it is not yet defined because the student has not yet finished Math 101. We can do this in Python like so:

First, we create a `namedtuple` to represent a student:

```py
from collections import namedtuple

# Create a class called student
student = namedtuple("student", ["name", "grade", "course_number"]) 
```
We then create records for the sample data:

```py
# Create the records for the students in the form of tuples

peter = student("Peter", 'B', 101)
amanda = student("Amanda", 'C', 101 )
sarah = student("Sarah", 'A', 102)
lisa = student("Lisa", 'D', 101)
alex = student("Alex", 'A', 102)
maria = student("Maria", 'B', 101)
andrew = student("Andrew", 'C', 102)

math_class = (peter, amanda, sarah, lisa, alex, maria, andrew)
```

Finally, we use `map()` to create a new record for a student consisting of:
* their name
* the initial grade X
* the math course Math 201 only IF the student's previous grade was B or above in their respective math course

```py
math_201 = map(lambda s: student(s.name, 'X', 201), filter(lambda q: q.grade <= 'B', math_class))

print(tuple(math_201))
```

The line `q.grade <= 'B'` removes students with grades less than B. The `<=` appears because the character A is lower than B in the alphabet (how Python compares the characters).

The code  "maps a filtered value". Remember, that because `map()` returns an iterable, we use the `tuple()` function to generate all the values we need.

The output will be all the students who are eligible for Math 201:
```
(student("Peter, 'X', 201), student("Sarah",' X', 201), student("Alex",  'X', 201), student("Maria",  'X', 201))
```