Use pandas to sort and filter the rows of a dataset.

Sorting and Filtering Rows

Sometimes, we're only interested in looking at certain rows and columns of a DataFrame. We can select sections of a DataFrame using two methods: `.loc[]` and `.iloc[]`.

We'll explore how both of these work using the following mini-DataFrame `df`.

|| Make | Year |
|---| --- | --- |
|0| Ford | 2017 |
|1| Ford | 2022 |
|2| Toyota | 2019 |
|3| Toyota | 2020 |

Before demonstrating `.loc[]` and `.iloc[]`, we'll sort `df` by `Year`. This isn't necessary to use `.loc[]` or `.iloc[]`, but it will make the difference between them clearer.

|| Make | Year |
|---| --- | --- |
|1| Ford | 2022 |
|3| Toyota | 2020 |
|2| Toyota | 2019 |
|0| Ford | 2017 |

Note that the index is now out-of-order. This will be important later on.

## Using `.loc[]`

The `.loc[]` index works using the index labels, and the syntax is

```py
df.loc[list_of_row_labels, list_of_column_labels]
```

For example, the code
```py
df.loc[[0,3],['Make']]
```

would perform the following selection:

<table>
<tr><th>Full DataFrame</th><th>Selection</th></tr>
<tr><td>

|| Make | Year |
|---| --- | --- |
|1| Ford | 2022 |
|3| Toyota | 2020 |
|2| Toyota | 2019 |
|0| Ford | 2017 |

</td><td>

|| Make |
|---| --- |
|0| Ford |
|3| Toyota |

</td></tr> </table>

The first list (`[0,3]`) instructs `.loc[]` to select the rows labelled `0` and `3` in order. The second list (`['Make']`) instructs `.loc[]` to select only the column labelled `Make`.

## Using `.iloc[]`

The `.iloc[]` index works using the *position* of the rows and columns, instead of their labels. The syntax is essentially the same, except that the lists contain positions instead of labels:

```py
df.iloc[list_of_row_positions, list_of_column_positions]
```

where

- both the row positions and column positions start at `0`
- Rows are counted top to bottom
- columns are counted from left to right 

For example, the code

```py
df.iloc[[0,3],[0]]
```

performs the selection

<table>
<tr><th>Full DataFrame</th><th>Selection</th></tr>
<tr><td>

|| Make | Year |
|---| --- | --- |
|1| Ford | 2022 |
|3| Toyota | 2020 |
|2| Toyota | 2019 |
|0| Ford | 2017 |

</td><td>

|| Make |
|---| --- |
|1| Ford |
|0| Ford |

</td></tr> </table>

Here, the list `[0,3]` instructs `.iloc` to select only the first row (position `0`) and the fourth row (position `3`). The column list `[0]` instructs `.iloc` to select the first column, which is the column labelled `Make`.

It is very common to make what are called `off-by-one` errors by starting positions at `1` instead of `0`. For example, it's natural to write the list `[1,4]` to select the first and fourth rows, instead of the list `[0,3]`. Be patient, and remember to check for this if you're receiving the wrong selection from `.iloc[]`.


Selecting Specific Rows

Instead of specifying lists of rows and columns with `.loc/.iloc`, we can specify a **slice**, which is like saying `select all rows from row A to row B`.

## Slices with `.iloc`

If we want to select the first five rows of a DataFrame using `iloc`, we could use the list of row positions `[0,1,2,3,4]`. 

A slice reproduces a consecutive list by specifying where to start and stop, using the syntax

```py
start_position:stop_position
```

where
- `start_position` is the first position to include
- `stop_position` is one position *beyond* the last position to include

For example, to reproduce our list `[0,1,2,3,4]`, we would use the slice `0:5` to indicate:

- start at position `0`
- stop before position `5`
- include all positions in-between (`1`, `2`, `3`, `4`)

Let's explore a practical example, using our `vehicles` DataFrame. The relevant section of the DataFrame is below:

||id   |model                        |year|transmission|
|--|-----|-----------------------------|----|------------|
|1940|12988|amg e53 4matic+ (convertible)|2022|auto        |
|718|689  |avalanche ffv                |2007|auto        |
|1663|950  |impala                       |2010|auto        |
|1581|250  |yukon xl ffv                 |2004|auto        |

Suppose we want to select the first three rows and the `model`, `year`, and `transmission` columns. We will need two slices, one for the rows and one for the columns:

- *rows*: we want to start at the first row (position `0`) and end *before* the fourth row (position `3`). The slice is `0:3` and will include positions `0`, `1`, and `2`.
- *columns*: we want to start at the second column (position `1`) and end *before* the fifth column (position `4`). The slice is `1:4` and will include positions `1`, `2`, and `3`

The full syntax would be

```py
df.iloc[0:3,1:4]
```

## Slices with `.loc`

Unlike `.iloc`, slices using `.loc` include the label after the `:` in the slice syntax. In other words,

```py
start_label:stop_label
```

will include

- the row/column labeled `start_label`
- the row/column labeled `stop_label`
- all rows/columns in between

Let's use `.loc` to select the same section of `vehicles`: the first three rows and the columns from `model` to `transmission`. 

Again, we'll need two slices:

- *rows*: the first three rows start at the label `1940` and end at the end `1663`. The slice is `1940:1663` and will include rows labeled `1940`, `718`, and `1663`. Note how the `stop_label` is included, unlike with `.iloc`.
- *columns*: the first column to include is labeled `model` and the last is `transmission`. The slice is `'model':'transmission'` and will include columns labeled `model`, `year`, and `transmission`.

The full syntax would be

```py
df.loc[1940:1663,'model':'transmission']
```

## Open-Ended Slices

If we don't specify a start value, the slice will start at the first position/label. For example, the `.iloc` slice `:3` would include rows at positions `0`, `1`, and `2`.

Similarly, if we don't specify an end value, the slice will end at the final position/label. For example, the `.loc` slice `'year':` will include every column starting at `year` through to the final column of the DataFrame.

If we specify neither the start nor the final value, the slice will include all rows and columns. For example, the code

```py
vehicles.iloc[0:2,:]
```

will include rows at positions `0` and `1` along with every column.


Selecting Ranges of Rows

A Boolean mask tells us which rows have a certain property. What we usually want is to actually filter the DataFrame down to only those rows.

For example, in the last exercise we created a Boolean Mask 

```py
is_2023 = (df['Year'] == 2023)
```

which generated a column of `True`/`False` based on the original DataFrame:

<table>
<tr>
<th>

DataFrame `df`

</th><th>

`is_2023`

</th></tr>
<tr><td>

| year | manufacturer |
| --- | --- |
| 2023 | Chevrolet |
| 2020 | Chevrolet |
| 2023 | Honda |
| 2022| Honda |

</td><td>

| year |
| --- |
| True |
| False |
| True |
| False|

</td></tr> </table>

To select only the rows where this mask is `True`, we pass the mask to the DataFrame:

```py
df[is_2023]
```

Only the `True` rows from `is_2023` will be selected:

<table>
<tr>
<th>

DataFrame `df`

</th><th>

`df[is_2023]`

</th></tr>
<tr><td>

| year | manufacturer |
| --- | --- |
| 2023 | Chevrolet |
| 2020 | Chevrolet |
| 2023 | Honda |
| 2022| Honda |

</td><td>

| year | manufacturer |
| --- | --- |
| 2023 | Chevrolet |
| 2023 | Honda |

</td></tr> </table>

Note that it isn't necessary to assign the Boolean mask to its own variable. We can pass the comparison directly to the DataFrame:

```py
df[df['year'] == 2023]
```

While this make the code shorter and more efficient, it can also decrease readability (especially when comparisons become more complex.) 

For now, we'll use variables to help clarify the structure of the Boolean filters as we build them step-by-step. As you feel more comfortable constructing filters, you can start removing such intermediary steps.

Filtering Rows with Booleans

Data questions often require us to select data by properties, instead of by position/index. For example, we might want to select all rows in `vehicles` corresponding to the year `2023`.

If we were filtering the dataset by hand, we might go through the dataset looking at each row and comparing the `Year` value to `2023`. If they match, we keep the row. If they don't, we discard it.

We can automate this process in Pandas, using Python's **comparison operators** to compare values.

## Comparison Operators

An **operator** is a way of combining two variables to produce some output. In mathematics, for example, addition is an operator for combining two numbers: `3+4` combines `3` and `4`  to output `7`.

**Comparison operators** are operators that compare two variables and output `True` or `False`. In mathematics, `=` is a comparison operator: `3=4` is `False`, while `3=3` is `True`.

Python has all the same comparison operators we know from mathematics:
- `==` is Python's version of equality
- `!=` is Python's version of `not equal to`
- `<` is Python's version of `less than`
- `>` is Python's version of `greater than`
- `<=` is Python's version of `less than or equal`
- `>=` is Python's version of `greater than or equal`

Whenever we compare two variables using a comparison operator, Python returns a `True` if the comparison is correct, and a `False` if it is incorrect.

For example, here are a few comparisons and their output:

```py
3 == 4
# output: False
```

```py
3 < 4
# output: True
```

We can also use `==` to compare non-numeric variable types:

```py
'car' == 'truck'
# output: False
```

## Comparison Operators in Pandas

Pandas can perform comparisons for an entire column at once using the same comparisons. For example, to determine which rows of a `Year` column contain `2023`, we would use the code

```py
df['Year'] == 2023
```

The output of this code is a Series with `True` for all the rows where the comparison is `True`:

<table>
<tr>
<th>

DataFrame `df`

</th><th>

`df['Year']==2023`

</th></tr>
<tr><td>

| year | manufacturer |
| --- | --- |
| 2023 | Chevrolet |
| 2020 | Chevrolet |
| 2023 | Honda |
| 2022| Honda |

</td><td>

| year |
| --- |
| True |
| False |
| True |
| False|

</td></tr> </table>

The output True/False Series is called a **Boolean mask**. One way to think about this terminology is that a Boolean mask allows you to see only certain features of a column (e.g. whether the value is `2023`) while obscuring others.

Since a Boolean mask is technically just a Series with Boolean values, any DataFrame/Series methods we've used before will work on a Boolean mask (e.g. `.head()`).

It can be helpful to assign a Boolean mask to a variable. For example, let's assign the Boolean mask above to the variable `is_2023`:

```py
is_2023 = df['year'] == 2023
```

Note that the single `=` is what performs the variable assignment, and the value being assigned is the entire comparison `df['year'] == 2023`. 

Two `=` signs performs the comparison, and one `=` performs the assignment to the variable. 

It's good practice, though not strictly necessary, to use parentheses to make this clearer:

```py
is_2023 = (df['year'] == 2023)
```

In the next exercise, we'll use Boolean masks to filter DataFrames. For now, let's practice working with comparison operators!


Boolean Masks

Our vehicle dataset has more than one type of electric vehicle:

1. electric
2. hybrid electric
3. plug-in hybrid electric

To analyze all electric vehicles, we might need to select rows that have any of these three.

## Boolean operators: or

In English, we might say `this vehicle is electric or hybrid electric`. This statement is

- true if the vehicle is electric (the first property is true)
- true if the vehicle is hybrid electric (the second property is true)
- false only when the vehicle is neither electric nor hybrid electric

In Python, `or` behaves the same way: it outputs `True` as long as one of the combined comparisons is `True`.

For example, we can combine the comparison `'electric' == 'electric'` with the comparison `'biodiesel' == 'hybrid electric'`:

```py
('electric' == 'electric') or ('biodiesel' == 'hybrid electric')
# Output: True, since the first comparison is True
```

On the other hand, combining `'propane' == 'electric'` with `'biodiesel' == 'hybrid electric'` will be false, since neither comparison is true:

```py
('propane' == 'electric') or ('biodiesel' == 'hybrid electric')
> False
```

Pandas uses the symbol `|` instead of `or`. Take the following mini DataFrame `df`:

| fuel |
| --- |
| electric |
| hybrid electric |
| plug-in hybrid electric |
| propane |

Let's construct a Boolean mask that is `True` whenever `fuel` is `electric` or `hybrid electric`:

```py
is_electric = df['fuel'] == 'electric'
is_hybrid = df['fuel'] == 'hybrid electric'
combined = is_electric | is_hybrid
```

The first two rows will be `True`, since each matches one of the comparisons. The last two, on the other hand, will be false.

| fuel | combined output
| --- | --- |
| electric | True|
| hybrid electric | True |
| plug-in hybrid electric | False |
| propane | False|

Note that we can combine more than two options using `|`. For example, to include `plug-in hybrid electric`, we would add it to the comparisons:

```py
is_electric = df['fuel'] == 'electric'
is_hybrid = df['fuel'] == 'hybrid electric'
is_plugin = df['fuel'] == 'plug-in hybrid electric'
combined = is_electric | is_hybrid | is_plugin
```

Lastly, we can filter the DataFrame down to only `True` rows by passing `combined` to `df`:

```py
df[combined]
```

or by constructing the combined comparison within the brackets:

```py
df[is_electric | is_hybrid | is_plugin]
```

In either case, we'd receive the filtered DataFrame

| fuel |
| --- |
| electric |
| hybrid electric |
| plug-in hybrid electric |


Combining Booleans with Or

In the last exercise, we created a Boolean filter to find any rows involving electric fuel:

```py
is_electric = df['fuel'] == 'electric'
is_hybrid = df['fuel'] == 'hybrid electric'
is_plugin = df['fuel'] == 'plug-in hybrid electric'
electric = is_electric | is_hybrid | is_plugin
```

What if we wanted all **non**electric vehicles? We could create a new statement using multiple `|`s. But it would be easier to use our pre-created filter.

Pandas uses the symbol `~` for `not/non`. In our example, we could create a new Boolean mask `not_electric` using the pre-existing mask:

```py
not_electric = ~electric
```

The new filter `not_electric` will only contain those rows that are neither `electric` nor `hybrid electric` nor `plug-in hybrid electric`:

| fuel | electric | ~electric |
| --- | --- | --- |
| electric | True| False |
| hybrid electric | True | False |
| plug-in hybrid electric | True | False |
| propane | False| True |

Essentially, `~electric` simply swaps `True` and `False`:

- anywhere that `electric` is `True`, `~electric` is `False`
- anywhere that `electric` is `False`, `~electric` is `True`


Inverting Booleans with Not

Both rows and columns in a DataFrame have labels called **indexes**. In the vehicles DataFrame

- rows are labelled by numbers (0, 1, 2, etc.) 
- columns are labelled with text (`id`, `model`, `year`, etc.)

If we talk about "_the_ index" of a DataFrame, we'll always be referring to the row index.

## Default Index

By default, DataFrames come with a **RangeIndex** where the first row is labelled `0`, the second row is labelled `1`, and so on. 

To confirm this, we can call the `.index` attribute on a DataFrame:

```py
df.index
```

If the DataFrame has the default index, the output will be

```
RangeIndex(start=0,
    stop=number of rows,
    step=1)
```

indicating that the row labels 

- start at 0
- increase by 1 for each row
- end _before_ reaching the total number of rows (since we started counting at 0)

For example, a DataFrame with 3 rows might have the index `RangeIndex(start=0, stop=3, step=1)`.

In this RangeIndex, we start labelling rows at 0, stop before 3, and increase by 1 each row:

- the first row has the label 0
- the second row has the label 1
- the third row has the label 2

## Other Kinds of Index

Both DataFrames and Series (individual columns) can have other types of index. For example, let's call `.value_counts()` on `vehicle_type` in our vehicles DataFrame:

```py
vehicle_counts = vehicles['vehicle_type'].value_counts()
vehicle_counts
```

The output we'll receive is

```
sedan/wagon                  1205
suv                           742
pickup                        373
van                           189
passenger van/shuttle bus       7
vocational/cab chassis          4
```

Here, each row is labeled with the vehicle type as text. If we call

```py
vehicle_counts.index
```

we'll receive the output

```py
Index(['sedan/wagon', 'suv', 'pickup', 'van', 'passenger van/shuttle bus',
       'vocational/cab chassis'],
      dtype='object')
```

Unlike a RangeIndex, this is an Index consisting of a list of distinct objects/strings.

## Resetting the Index

It is possible to restore a standard `RangeIndex` by calling the method `.reset_index()`. For example, if we call

```py
vehicle_counts.reset_index()
```

we'll receive the following output

|| index |vehicle_type|
|---|---|---|
|0|	sedan/wagon|	1205|
|1|	suv|	742|
|2|	pickup|	373|
|3|	van	|189|
|4|	passenger van/shuttle bus|7|
|5|	vocational/cab chassis|	4|

Note that the column `index` in the `.reset_index()` DataFrame is no longer the index: it is a column containing the old index values pre-reset. The index after calling `.reset_index()` is the RangeIndex from `0` to `5`.

---

*How to Use Your Jupyter Notebook:*

- *You can run a cell in the Notebook to the right by placing your cursor in the cell and clicking the `Run` button or the `Shift`+`Enter/Return` keys.*
- *When you are ready to evaluate the code in your Notebook, press the `Save` button at the top of the Notebook or use the `control/command`+`s` keys before clicking the `Test Work` button at the bottom. Be sure to save your solution code in the cell marked `## YOUR SOLUTION HERE ##` or it will not be evaluated.*
- *When you are ready to move on, click Next.*

![Screenshot of the buttons at the top of a Jupyter Notebook. The Run and Save buttons are highlighted](https://static-assets.codecademy.com/Paths/ds-python/jupyter-buttons.png)

Row Indexes

Exploring datasets is easier if the rows have a predictable structure. For example, the vehicles dataset would be simpler to inspect if we knew that the models were listed in chronological order.

The syntax to sort a DataFrame `df` in pandas by a particular column is

```py
df = df.sort_values(
  by = 'Column'
)
```

where

- `.sort_values()` is a DataFrame method
- `by` is a keyword that receives the label of the column to sort by

By default, `.sort_values()` will sort from lowest to highest (or, for text, from A to Z). Passing `False` to the `ascending` parameter will sort from highest to lowest (or Z to A):

```py
df = df.sort_values(
  by = 'Column',
  ascending = False
)
```

Here's an example of calling

```py
.sort_values(by='year',
    ascending=False)
```

on a small DataFrame of vehicle models:

<table>
<tr><th>Original</th><th>Sorted</th></tr>
<tr><td>

| Make | Year |
| --- | --- |
| Ford | 2022 |
| Toyota | 2019 |
| Ford | 2017 |
| Toyota | 2020 |

</td><td>

| Make | Year |
| --- | --- |
| Ford | 2022 |
| Toyota | 2020 |
| Toyota | 2019 |
| Ford | 2017 |

</td></tr> </table>

Notice how the rows have been re-ordered to have the most recent model at the top, and the oldest at the bottom.

## Sorting an index

Calling `.sort_values()` will only sort values by a column. To sort rows by the *index* values instead, use `.sort_index()`. There's no need to specify a `by` parameter, since the method will already sort by the index. 

Like `.sort_values()`, passing `True/False` to `ascending=` controls the order of the index sort.

---

*How to Use Your Jupyter Notebook:*

- *You can run a cell in the Notebook to the right by placing your cursor in the cell and clicking the `Run` button or the `Shift`+`Enter/Return` keys.*
- *When you are ready to evaluate the code in your Notebook, press the `Save` button at the top of the Notebook or use the `control/command`+`s` keys before clicking the `Test Work` button at the bottom. Be sure to save your solution code in the cell marked `## YOUR SOLUTION HERE ##` or it will not be evaluated.*
- *When you are ready to move on, click Next.*

![Screenshot of the buttons at the top of a Jupyter Notebook. The Run and Save buttons are highlighted](https://static-assets.codecademy.com/Paths/ds-python/jupyter-buttons.png)

Sorting Rows

Congratulations! We've covered a lot in this lesson, including

- sorting DataFrames by columns and index
- selecting rows from DataFrames using labels and position
- comparing variable values using Python comparison operators
- creating Boolean masks using pandas and comparison operators
- combining comparisons using Boolean operators
- filtering a DataFrame using a Boolean mask

We've loaded a notebook with additional exercises to the right. When you're ready to move on, select `Up Next`!


Let's Review

The more rows a dataset has, the harder it can be to find the specific data we want. For example, in the notebook to the right we've loaded a dataset of 2500 vehicle models that use non-standard fuels like electricity (from [the US Department of Energy](https://afdc.energy.gov/data_download)). 

These models are listed in no particular order. So if we want to find out which vehicle has the best mileage, we'll need to sort the rows to make that information easily accessible.

If we want to look at only electric vehicles, we'll need to filter the rows to display only the relevant electric models.

In this lesson, you'll learn how to

- access specific rows of a DataFrame
- structure a DataFrame by sorting the rows
- filter a DataFrame's rows to zoom in on key categories

Along the way, you'll answer questions like
- which vehicles have the best mileage?
- are there fewer manual transmission vehicles over time?
- did the number of electric vehicle models increase after the Paris Agreement?

Feel free to explore the dataset in the notebook. When you're ready to learn more about rows, select `Next`!


Welcome to Sorting and Filtering Rows

The US Federal Government created an electric vehicle tax credit in 2010. What if we wanted a DataFrame of all electric vehicle models introduced after 2010? 

We would need to combine two different comparisons:

```py
vehicles['Year'] > 2010
```

and

```py
vehicles['fuel'] == 'electric'
```

Boolean operators allow us to combine comparisons into a single True/False value.

## Boolean operators: and

In English, we might say `This vehicle is electric and it was introduced after 2010`. The word `and` indicates that the statement is true only if both components are true:

1. the vehicle is indeed electric
2. the vehicle was in fact introduced after 2010

Python behaves exactly the same way: two comparisons can be combined with `and` to test if both are simultaneously true. 

For example, we can combine `2011 > 2010` with `'electric' == 'electric'`, using parentheses `()` to mark the two components for readability:

```py
(2011 > 2010) and ('electric' == 'electric')
# Output: True
```

On the other hand, combining `2011 > 2010` with `'biodiesel' == 'electric'` produces `False` since the second comparison is false:

```py
(2011 > 2010) and ('biodiesel' == 'electric')
> False
```

### And in pandas

Pandas also provides a version of `and`, although pandas uses the symbol `&`. Let's perform our combined comparison on the following mini DataFrame df:

| year | fuel |
| --- | --- |
| 2009 | electric |
| 2010| hybrid electric |
| 2011 | electric |
| 2012 | propane |

While we can combine the comparisons on a single line, it is more readable to do this piece-by-piece:

```py
is_electric = df['fuel'] == 'electric'
year_gt2010 = df['year'] > 2010
combined = is_electric & year_gt2010
```

Here's the output `combined` next to the original DataFrame:

<table>
<tr><th>Original DataFrame</th><th>Combined Mask</th></tr>
<tr><td>

| year | fuel |
| --- | --- |
| 2009 | electric |
| 2010| hybrid electric |
| 2011 | electric |
| 2012 | propane |

</td><td>

| output|
| --- |
| False |
| False |
| True |
| False |

</td></tr> </table>

Lastly, we can select only the `True` rows from the DataFrame

```py
df[combined]
```

producing

| year | fuel |
| --- | --- |
| 2011 | electric |

Usually, instead of creating a `combined` mask, we just combine the individual masks when we pass them to the DataFrame:

```py
df[is_electric & year_gt2010]
```

Technically, we can perform the entire process within the `[]`, using parentheses to separate the individual comparisons:

```py
df[(df['fuel']=='electric') & (df['year'] > 2010)]
```

But this can very quickly become challenging to read and debug. Unless we're particularly worried about efficiency, it can be helpful to use "extra" variables to build the filter one step at a time.

Lastly, it is possible for there to be no rows where a Boolean mask is `True`. In this case, filtering the DataFrame will produce a result with column labels in the header but no actual rows. This doesn't mean there's a mistake in the code (necessarily), only that no rows match the criteria we are filtering for!
