ENDS 12/4: Get 50% off your Pro annual membership using code [CYBER23](https://www.codecademy.com/checkout?periods=year&plan_id=proGoldAnnualV2&discountCode=CYBER23)


Given a set of data points, find a line that fits the data best! This is the simplest form of regression, and allows us to make predictions of future points.

Linear Regression

The goal of a linear regression model is to find the slope and intercept pair that minimizes loss on average across all of the data.

The interactive visualization in the browser lets you try to find the line of best fit for a random set of data points:
- The slider on the left controls the `m` (slope)
- The slider on the right controls the `b` (intercept)

You can see the total loss on the right side of the visualization. To get the line of best fit, we want this loss to be as small as possible.

To check if you got the best line, check the "Plot Best-Fit" box.

Randomize a new set of points and try to fit a new line by entering the number of points you want (try `8`!) in the textbox and pressing <kbd>Randomize Points</kbd>.

Minimizing Loss

How do we know when we should stop changing the parameters `m` and `b`? How will we know when our program has learned enough?

To answer this, we have to define convergence. **Convergence** is when the loss stops changing (or changes very slowly) when parameters are changed.

Hopefully, the algorithm will converge at the best values for the parameters `m` and `b`.


Convergence

When we think about how we can assign a slope and intercept to fit a set of points, we have to define what the _best fit_ is.

For each data point, we calculate **loss**, a number that measures how bad the model's (in this case, the line's) prediction was. You may have seen this being referred to as error.

We can think about loss as the squared distance from the point to the line. We do the squared distance (instead of just the distance) so that points above and below the line both contribute to total loss in the same way:

<img src="https://content.codecademy.com/programs/machine-learning/linear-regression/points.svg">

In this example:
- For point A, the squared distance is `9` (3²)
- For point B, the squared distance is `1` (1²)

So the total loss, with this model, is `10`. If we found a line that had less loss than `10`, that line would be a better model for this data.

Loss

We have constructed a way to find the "best" `b` and `m` values using gradient descent! Let's try this on the set of baseball players' heights and weights that we saw at the beginning of the lesson.

Use Your Functions on Real Data

We have a function to find the gradient of `b` at every point. To find the `m` gradient, or the way the loss changes as the slope of our line changes, we can use this formula:

```tex
-\frac{2}{N}\sum_{i=1}^{N}x_i(y_i-(mx_i+b))
```

Once more:
- `N` is the number of points you have in your dataset
- `m` is the current gradient guess
- `b` is the current intercept guess

To find the `m` gradient:
- we find the sum of `x_value * (y_value - (m*x_value + b))` for all the `y_value`s and `x_value`s we have
- and then we multiply the sum by a factor of `-2/N`. `N` is the number of points we have.

Once we have a way to calculate both the `m` gradient and the `b` gradient, we'll be able to follow both of those gradients downwards to the point of lowest loss for both the `m` value and the `b` value. Then, we'll have the best `m` and the best `b` to fit our data! 


Gradient Descent for Slope

The purpose of machine learning is often to create a model that explains some real-world data, so that we can predict what may happen next, with different inputs.

The simplest model that we can fit to data is a line.  When we are trying to find a line that fits a set of data best, we are performing **Linear Regression**.

We often want to find lines to fit data, so that we can predict unknowns. For example:

- The market price of a house vs. the square footage of a house. Can we predict how much a house will sell for, given its size?
- The tax rate of a country vs. its GDP. Can we predict taxation based on a country's GDP?
- The amount of chips left in the bag vs. number of chips taken. Can we predict how much longer this bag of chips will last, given how much people at this party have been eating?

Imagine that we had this set of weights plotted against heights of a large set of professional baseball players:

![heights vs weights](https://content.codecademy.com/programs/data-science-path/linear_regression/weight_height.png)

To create a linear model to explain this data, we might draw this line:

![heights vs weights](https://content.codecademy.com/programs/data-science-path/linear_regression/weight_height_line.png)

Now, if we wanted to estimate the weight of a player with a height of 73 inches, we could estimate that it is around 143 pounds.

A line is a rough approximation, but it allows us the ability to explain and predict variables that have a linear relationship with each other. In the rest of the lesson, we will learn how to perform Linear Regression.

Introduction to Linear Regression

Now that we know how to calculate the gradient, we want to take a "step" in that direction. However, it's important to think about whether that step is too big or too small. We don't want to overshoot the minimum error!

We can scale the size of the step by multiplying the gradient by a *learning rate*.

To find a new `b` value, we would say:

```py
new_b = current_b - (learning_rate * b_gradient)
```

where `current_b` is our guess for what the `b` value is, `b_gradient` is the gradient of the loss curve at our current guess, and `learning_rate` is proportional to the size of the step we want to take.

In a few exercises, we'll talk about the implications of a large or small learning rate, but for now, let's use a fairly small value.

Put it Together

At each step, we know how to calculate the gradient and move in that direction with a step size proportional to our learning rate. Now, we want to make these steps until we reach convergence. 


Put it Together II

As we try to minimize loss, we take each parameter we are changing, and move it as long as we are decreasing loss. It's like we are moving down a hill, and stop once we reach the bottom:

<img src=https://content.codecademy.com/programs/data-science-path/linear_regression/loss_curve.svg>


The process by which we do this is called **gradient descent**. We move in the direction that decreases our loss the most. _Gradient_ refers to the slope of the curve at any point. 

For example, let's say we are trying to find the intercept for a line. We currently have a guess of `10` for the intercept. At the point of `10` on the curve, the slope is downward. Therefore, if we increase the intercept, we should be lowering the loss. So we follow the gradient downwards.

<img src=https://content.codecademy.com/programs/data-science-path/linear_regression/Linear_regression_gif_1.gif>

We derive these gradients using calculus. It is not crucial to understand how we arrive at the gradient equation. To find the gradient of loss as intercept changes, the formula comes out to be:


```tex
-\frac{2}{N}\sum_{i=1}^{N}(y_i-(mx_i+b))
```

- `N` is the number of points we have in our dataset
- `m` is the current gradient guess
- `b` is the current intercept guess

Basically:

- we find the sum of `y_value - (m*x_value + b)` for all the `y_value`s and `x_value`s we have
- and then we multiply the sum by a factor of `-2/N`. `N` is the number of points we have.


Gradient Descent for Intercept

Congratulations! You've now built a linear regression algorithm from scratch.

Luckily, we don't have to do this every time we want to use linear regression. We can use Python's scikit-learn library. Scikit-learn, or `sklearn`, is used specifically for Machine Learning. Inside the `linear_model` module, there is a `LinearRegression()` function we can use:

```py
from sklearn.linear_model import LinearRegression
```

You can first create a `LinearRegression` model, and then fit it to your `x` and `y` data:

```py
line_fitter = LinearRegression()
line_fitter.fit(X, y)
```

The `.fit()` method gives the model two variables that are useful to us:
1. the `line_fitter.coef_`, which contains the slope
2. the `line_fitter.intercept_`, which contains the intercept

We can also use the `.predict()` function to pass in x-values and receive the y-values that this line would predict:

```py
y_predicted = line_fitter.predict(X)
```

**Note:** the `num_iterations` and the `learning_rate` that you learned about in your own implementation have default values within scikit-learn, so you don't need to worry about setting them specifically!

Scikit-Learn

We have seen how to implement a linear regression algorithm in Python, and how to use the linear regression model from scikit-learn. We learned:

- We can measure how well a line fits by measuring _loss_.
- The goal of linear regression is to minimize loss.
- To find the line of best fit, we try to find the `b` value (intercept) and the `m` value (slope) that minimize loss.
- _Convergence_ refers to when the parameters stop changing with each iteration.
- _Learning rate_ refers to how much the parameters are changed on each iteration.
- We can use Scikit-learn's `LinearRegression()` model to perform linear regression on a set of points.

These are important tools to have in your toolkit as you continue your exploration of data science. 



Review

In the last exercise, you were probably able to make a rough estimate about the next data point for Sandra's lemonade stand without thinking too hard about it. For our program to make the same level of guess, we have to determine what a line would look like through those data points.

A line is determined by its _slope_ and its _intercept_. In other words, for each point `y` on a line we can say:

```tex
y = m x + b
```
where `m` is the slope, and `b` is the intercept. `y` is a given point on the y-axis, and it corresponds to a given `x` on the x-axis.

The slope is a measure of how steep the line is, while the intercept is a measure of where the line hits the y-axis.

When we perform Linear Regression, the goal is to get the "best" `m` and `b` for our data. We will determine what "best" means in the next exercises.

Points and Lines

We want our program to be able to iteratively _learn_ what the best `m` and `b` values are. So for each `m` and `b` pair that we guess, we want to move them in the direction of the gradients we've calculated. But how far do we move in that direction?

We have to choose a **learning rate**, which will determine how far down the loss curve we go.

A small learning rate will take a long time to converge &mdash; you might run out of time or cycles before getting an answer. A large learning rate might skip over the best value. It might _never_ converge! Oh no!

<img src=https://content.codecademy.com/programs/data-science-path/linear_regression/Linear_regression_gif_2.gif>

Finding the absolute best learning rate is not necessary for training a model. You just have to find a learning rate large enough that gradient descent converges with the efficiency you need, and not so large that convergence never happens.

