Learn how to explore a dataset by calculating summary statistics and creating basic data visualizations for each of the features.

Data Summaries

While summary statistics are certainly helpful for exploring and quantifying a feature, we might find it hard to wrap our minds around a bunch of numbers. This is why data visualization is such a powerful element of EDA.

For quantitative variables, _boxplots_ and _histograms_ are two common visualizations. These plots are useful because they simultaneously communicate information about minimum and maximum values, central location, and spread. Histograms can additionally illuminate patterns that can impact an analysis (e.g., skew or multimodality).

Python's `seaborn` library, built on top of `matplotlib`, offers the [`boxplot()`](https://seaborn.pydata.org/generated/seaborn.boxplot.html) and [`histplot()`](https://seaborn.pydata.org/generated/seaborn.histplot.html) functions to easily plot data from a `pandas` DataFrame:

```python
import matplotlib.pyplot as plt 
import seaborn as sns

# Boxplot for rent
sns.boxplot(x='rent', data=rentals)
plt.show()
plt.close()
```

![boxplot of rent](https://static-assets.codecademy.com/Courses/EDA/Data_Summaries/boxplot.svg)

```python
# Histogram for rent
sns.histplot(x='rent', data=rentals)
plt.show()
plt.close()
```
![histogram of rent](https://static-assets.codecademy.com/Courses/EDA/Data_Summaries/histplot.svg)

Visualizing Quantitative Variables

When it comes to categorical variables, the measures of central tendency and spread that worked for describing numeric variables, like mean and standard deviation, generally becomes unsuitable when we're dealing with discrete values. Unlike numbers, categorical values are not continuous and oftentimes do not have an intrinsic ordering. 

Instead, a good way to summarize categorical variables is to generate a frequency table containing the count of each distinct value. For example, we may be interested to know how many of the New York City rental listings are from each borough. Related, we can also find which borough has the most listings.

The `pandas` library offers the [`.value_counts()`](https://pandas.pydata.org/docs/reference/api/pandas.Series.value_counts.html) method for generating the counts of all values in a DataFrame column:

```python
# Counts of rental listings in each borough
df.borough.value_counts()
```

**Output**:

```pseudo
Manhattan    3539
Brooklyn     1013
Queens        448
```

By default, it returns the results sorted in descending order by count, where the top element is the _mode_, or the most frequently appearing value. In this case, the mode is `Manhattan` with 3,539 rental listings.

Value Counts for Categorical Data

For categorical variables, bar charts and pie charts are common options for visualizing the count (or proportion) of values in each category. They can also convey the relative frequencies of each category.

Python's `seaborn` library offers several functions that can create bar charts. The simplest for plotting the counts is [`countplot()`](https://seaborn.pydata.org/generated/seaborn.countplot.html):

```python
# Bar chart for borough
sns.countplot(x='borough', data=rentals)
plt.show()
plt.close()
```
![Bar chart of borough](https://static-assets.codecademy.com/Courses/EDA/Data_Summaries/countplot.svg)

There are currently no functions in the `seaborn` library for creating a pie chart, but the `pandas` library provides a convenient [wrapper function](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.plot.pie.html) around `matplotlib`'s  [`pie()`](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.pie.html) function that can generate a pie chart from any column in a DataFrame:

```python
# Pie chart for borough
rentals.borough.value_counts().plot.pie()
plt.show()
plt.close()
```

![Pie chart of borough](https://static-assets.codecademy.com/Courses/EDA/Data_Summaries/pieplot.svg)

In general, many data analysts avoid pie charts because people are better at visually comparing areas of rectangles than wedges of a pie. For a variable with a small number of categories (i.e., fewer than three), a pie chart is a reasonable choice; however, for more complex data, a bar chart is usually preferable.


Visualizing Categorical Variables

In this lesson, you've learned about the common ways to summarize and visualize quantitative and categorical variables for the purpose of EDA. 

- We can use `.describe(include='all')` to quickly display common summary statistics for all columns in a `pandas` DataFrame.
- For _quantitative_ variables, measures of central tendency (e.g., mean, median, mode) and spread (e.g., range, variance, standard deviation) are good ways to summarize the data. Boxplots and histograms are often used for visualization.
- For _categorical_ variables, the relative frequencies of each category can be summarized using a table of counts or proportions. Bar charts and pie charts are often used for visualization.

Being able to use the appropriate metrics and visuals to explore the variables in your dataset can help you to draw insights from your data and prepare for more rigorous analysis and modeling down the road.


This image shows different variants of bar charts and pie charts.

Review

A counts table is one approach for exploring categorical variables, but sometimes it is useful to also look at the proportion of values in each category. For example, knowing that there are 3,539 rental listings in Manhattan is hard to interpret without any context about the counts in the other categories. On the other hand, knowing that Manhattan listings make up 71% of all New York City listings tells us a lot more about the relative frequency of this category.

We can calculate the proportion for each category by dividing its count by the total number of values for that variable:

```python
# Proportions of rental listings in each borough
rentals.borough.value_counts() / len(rentals.borough)
```

**Output**:

```pseudo
Manhattan    0.7078
Brooklyn     0.2026
Queens       0.0896
```

Alternatively, we could also obtain the proportions by specifying `normalize=True` to the `.value_counts()` method:

```python
df.borough.value_counts(normalize=True)
```


Value Proportions for Categorical Data

The _spread_ of a quantitative variable describes the amount of variability. This is important because it provides context for measures of central tendency. For example, if there is a lot of variability in New York City rent prices, we can be less certain that the mean or median price is representative of what the typical rent is.

There are several common measures of spread:

- **Range**: The difference between the maximum and minimum values of a variable.
- **Interquartile range (IQR)**: The difference between the 75th and 25th percentile values.
- **Variance**: The average of the squared distance from each data point to the mean.
- **Standard deviation (SD)**: The square root of the variance.
- **Mean absolute deviation (MAD)**: The mean absolute value of the distance between each data point and the mean.

For our `rentals` DataFrame, we can calculate the spread for the `rent` column as follows:

```python
# Range
rentals.rent.max() - rentals.rent.min()

# Interquartile range
rentals.rent.quantile(0.75) - rentals.rent.quantile(0.25)

from scipy.stats import iqr
iqr(rentals.rent)  # alternative way

# Variance
rentals.rent.var()

# Standard deviation
rentals.rent.std()

# Mean absolute deviation
rentals.rent.mad()
```


Spread for Quantitative Data

For quantitative variables, we often want to describe the _central tendency_, or the "typical" value of a variable. For example, what is the typical cost of rent in New York City?

There are several common measures of central tendency:

- **Mean**: The average value of the variable, calculated as the sum of all values divided by the number of values.
- **Median**: The middle value of the variable when sorted.
- **Mode**: The most frequent value of the variable.
- **Trimmed mean**: The mean excluding x percent of the lowest and highest data points.

For our `rentals` DataFrame with a column named `rent` that contains rental prices, we can calculate the central tendency statistics listed above as follows:

```python
# Mean
rentals.rent.mean()

# Median
rentals.rent.median()

# Mode
rentals.rent.mode()

# Trimmed mean
from scipy.stats import trim_mean
trim_mean(rentals.rent, proportiontocut=0.1)  # trim extreme 10%
``` 


Central Tendency for Quantitative Data

Before diving into formal analysis with a dataset, it is often helpful to perform some initial investigations of the data through exploratory data analysis (EDA) to get a better sense of what you will be working with. Basic summary statistics and visualizations are important components of EDA as they allow us to condense a large amount of information into a small set of numbers or graphics that can be easily interpreted.

This lesson focuses on _univariate summaries_, where we explore each variable separately. This is useful for answering questions about each individual feature. Variables can typically be classified as _quantitative_ (i.e., numeric) or _categorical_ (i.e., discrete). Depending on its type, we may want to choose different summary metrics and visuals to use.

Let's say we have the following dataset on New York City rental listings imported into a `pandas` DataFrame (subsetted from the [StreetEasy dataset](https://www.codecademy.com/content-items/d19f2f770877c419fdbfa64ddcc16edc)):

```python
import pandas as pd

# Import dataset
rentals = pd.read_csv('streeteasy.csv')

# Preview first 5 rows
print(rentals.head())
```

| rent | size_sqft | borough |
|---|---|---|
| 2550 | 480 | Manhattan |
| 11500 | 2000 | Manhattan |
| 3000 | 1000 | Queens |
| 4500 | 916 | Manhattan |
| 4795 | 975 | Manhattan |

As seen, we have two quantitative variables (`rent` and `size_sqft`) and one categorical variable (`borough`). The `pandas` library offers a handy method [`.describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html) for displaying some of the most common summary statistics for the columns in a DataFrame. By default, the result only includes numeric columns, but we can specify `include='all'` to the method to display categorical ones as well:

```python
# Display summary statistics for all columns
print(rentals.describe(include='all'))
```

|| rent | size_sqft | borough |
|---|---|---|---|
| count | 5000.000000 | 5000.000000 | 5000 |
| unique | NaN | NaN | 3 |
| top | NaN | NaN | Manhattan |
| freq | NaN | NaN | 3539 |
| mean | 4536.920800 | 920.101400 | NaN |
| std | 2929.838953 | 440.150464 | NaN |
| min | 1250.000000 | 250.000000 | NaN |
| 25% | 2750.000000 | 633.000000 | NaN |
| 50% | 3600.000000 | 800.000000 | NaN |
| 75% | 5200.000000 | 1094.000000 | NaN |
| max | 20000.000000 | 4800.000000 | NaN |

This is a great way to get an overview of all the variables in a dataset. Notice how different statistics are displayed depending on the variable type. In the rest of the lesson, we'll look more closely at the common ways to summarize and visualize quantitative and categorical variables. 
