Get ready to dive deeper into logistic regression! Learn about the assumptions that go into it, what to do when there's class imbalance and how to set and evaluate prediction thresholds!

Logistic Regression II

We have examined how changing the threshold can affect the logistic regression predictions.  There is a continuum of predictions available in a single model by varying the threshold incrementally from zero to one.  For each of these thresholds, the True Positive Rate (TPR) and the False Positive Rate (FPR) can be calculated and then plot.  The resulting curve these points form is known as the Receiver Operating Characteristic (ROC) curve.

![There is a two-dimensional plot with "FPR" on the x-axis and "TPR" on the y-axis. Both axes range from 0 to 1. There is an orange curve on the plot that is positively sloped with the words ROC labeled above it. There are three data points on this ROC curve with the labels .25, .5, and .75. There is also a straight dashed blue diagonal line that starts at zero and has a slope of 1.](https://static-assets.codecademy.com/Paths/machine-learning-engineer-career-path/logistic-regression/ROC_curve_thresh.svg)

In the ROC curve plotted above, the True Positive Rate (`TPR = TP / TP + FN`) is on the y-axis and the False Positive Rate (`FPR = FP / TN + FP`) is on the x-axis. The ROC curve is the orange line and the dashed blue line is the Dummy Classification line, which is the equivalent of random guessing.

Notice there are three data points on the ROC curve, each labeled with their threshold values. The classification threshold of `.5` will give us a TPR of about `.65` with an FPR of about `.28`. For our specific data, we want a higher TPR so that we catch every malignant tumor. We might select a lower threshold of `.25` so that our TPR is about `.8`, even though this may give us an FPR of about `.4`. The ROC curve can help us decide on a threshold that best fits our specific classification problem. 

While the ROC curve measures the probabilities, the AUC (Area Under the Curve) gives us a single metric for separability. The AUC tells us how well our model can distinguish between the two classes. An AUC score close to 1 is a near-perfect classifier, whereas a value of 0.5 is equivalent to random guessing. To visualize different AUC scores, look at the ROC curve plots below:

![There are two plots in this image that both have "FPR" on the x-axis and "TPR" on the y-axis. The plot on the left has an orange line that is labeled "ROC" that goes straight up at 0 on the x-axis and then goes directly to the right at the top of the plot. There are words inside the plot that says "AUC = 1". The plot on the right has a straight orange line labeled "ROC" that starts at 0 and has a slope of 1 (making the line 45 degrees away from the x-axis). The words inside this plot state, "AUC = .5".](https://static-assets.codecademy.com/Paths/machine-learning-engineer-career-path/logistic-regression/ROC_AUC.svg)

ROC Curve and AUC

Class imbalance is when your binary classes for the outcome variable are not evenly split.  Technically, anything different from a 50/50 distribution would be imbalanced and need appropriate care. In the case of rare events, sometimes the positive class can be less than 1% of the total. If your classes are significantly imbalanced, this could create a bias towards the majority class since the model learns that it can have a higher accuracy if it predicts the majority class more often.

###### Positivity Rate
We can use the positivity rate to tell us how balanced our classes are. The positivity rate is the rate of occurrence for the positive class. With our breast cancer data, the formula is `Positivity Rate = Total Malignant Cases / Total Cases`. If our positivity rate is close to .5, then our classes are balanced.

###### Stratification
If your classes are imbalanced (more likely to happen with smaller datasets) then this difference can become even greater after you split your data into a training and testing dataset. One way to mitigate this is to randomly split using stratification on the class labels. Stratification is when the data is sorted into subgroups to ensure a nearly equal class distribution in your train and test sets. After using stratification, the training and testing datasets should have a very similar positivity rate (but stratification does not necessarily cause the positivity rate of the dataset to reach closer to .5).

###### Undersampling/Oversampling
To bring the positivity rate of the dataset closer to .5, we can undersample the majority class or oversample the minority class. For oversampling, repeated samples (with replacement) are taken from the minority class until the size is equal to that of the majority class. This causes the same data to be used multiple times, giving a higher weight to these samples. Alternatively, undersampling leaves out some of the majority class data to have the same number of samples as the minority class, leaving fewer data to build the model. We will not be practicing undersampling/oversampling in this exercise even though it can be a useful way to balance the classes.

###### Balance the Class Weight
When training a model, it is the default for every sample to be weighted equally. However, in the case of class imbalance, this can result in poor predictive power for the smaller of the two classes. A way to counteract this in logistic regression is to use the parameter `class_weight='balanced'`. This applies a weight inversely proportional to the class frequency, therefore supplying higher weight to misclassified instances in the smaller class. While overall accuracy may not increase, this can increase the accuracy for the smaller class (e.g. increase the number of malignant cases correctly diagnosed). 

Keep in mind that we want the recall score (also known as the True Positive Rate) to be as high as we can get it for our breast cancer data.
```
Recall = TP / TP + FN
```


Class Imbalance

We're now ready to delve deeper into Logistic Regression! In this lesson, we will cover the different assumptions that go into logistic regression, model hyperparameters, how to evaluate a classifier, ROC curves, and what to do when there's a class imbalance in the classification problem we're working with. 

For this lesson, we will be using the [Wisconsin Breast Cancer Data Set](https://www.kaggle.com/uciml/breast-cancer-wisconsin-data) (Diagnostic) to predict whether a tumor is benign (`0`) or malignant (`1`) based on characteristics of the cells, such as radius, texture, smoothness, etc.  Like a lot of real-world data sets, the distribution of outcomes is uneven (benign diagnoses are more common than malignant) and there is a bias in terms of the importance of the outcomes (classifying all malignant cases correctly is of the utmost importance). 

We're going to begin with the primary assumptions about the data that need to be checked before implementing a logistic regression model.

###### 1. The target variable is binary 
One of the most basic assumptions of logistic regression is that the outcome variable needs to be binary, which means there are two possible outcomes. Multinomial logistic regression is an exception to this assumption and is beyond the scope of this lesson.

###### 2. Independent observations
While often overlooked, checking for independent observations in a data set is important for logistic regression.  This can be violated if, in this case, patients are biopsied multiple times (repeated sampling of the same individual).  

###### 3. Large enough sample size
Since logistic regression is fit using maximum likelihood estimation instead of least squares minimization, there must be a large enough sample to get convergence. When a model fails to converge, this causes the estimates to be extremely inaccurate.  Now, what does a "large enough" sample mean? Often a rule of thumb is that there should be at least 10 samples per feature for the smallest class in the outcome variable. 

For example, if there were 100 samples and the outcome variable `diagnosis` had 60 benign tumors and 40 malignant tumors, then the max number of features allowed would be 4. To get 4 we took the smallest of the classes in the outcome variable, 40, and divided it by 10.

###### 4. No influential outliers
Logistic regression is sensitive to outliers, so we must remove any extremely influential outliers for model building.  Outliers are a broad topic with many different definitions -- z-scores, scaler of the interquartile range, Cook's distance/influence/leverage, etc -- so there are many ways to identify them.  But here, we will use visual tools to rule out obvious outliers.  

Assumptions of Logistic Regression I

##### 1. Features linearly related to log odds
Similar to linear regression, the underlying assumption of logistic regression is that the features are linearly related to the logit of the outcome.  To test this visually, we can use Seaborn's regplot, with the parameter `logistic= True` and the x value as our feature of interest.  If this condition is met, the fit model will resemble a sigmoidal curve (as is the case when `x=radius_mean`). 

We've added code to create another plot using the feature `fractal_dimension_mean`. Press **Run** in the workspace. How do the curves compare?

##### 2. Multicollinearity
Like in linear regression, one of the assumptions is that there is no multicollinearity in the data. Meaning the features should not be highly correlated. Multicollinearity can cause the coefficients and p-values to be inaccurate. With a correlation plot, we can see which features are highly correlated and then we can drop one of the features.

We're going to look at the "mean" features which are highly correlated with each other using a heatmap correlation plot.


Assumptions of Logistic Regression II

#### Model Training and Hyperparameters
Now that we have checked the assumptions of Logistic Regression, we can train and predict a model using `scikit-learn`.  We will first set the _hyperparameters_ of our model. Hyperparameters are set before the model implementation step and tuned later to improve model performance. Conversely, parameters are the result of model implementation, such as the intercept and coefficients.

_Note_: Within `scikit-learn` these hyperparameters are often referred to as "parameters" which might cause some confusion. It is worth noting that the meaning within `scikit-learn` documentation refers to these being "parameters" _of the function_ and not of the model itself.

#### Evaluation Metrics
Despite the name, logistic regression is being used as a classifier here, so any evaluation metrics for classification tasks will apply. The simplest metric is accuracy -- how many correct predictions did we make out of the total?  However, when classes are imbalanced, this can be a misleading metric for model performance.  Similarly, if we care more about accurately predicting a certain class, other metrics may be more appropriate to use, such as precision, recall, or F1-score may be better to evaluate performance. All of these metrics are available in `scikit-learn`. Check out the [Evaluation Metrics](https://www.codecademy.com/paths/machine-learning-engineer/tracks/mle-machine-learning-fundamentals/modules/mlecp-supervised-learning-i-regressors-classifiers-and-trees/lessons/mlfun-evaluation-metrics-classification/exercises/confusion-matrix) lesson if you'd like to brush up on the same.
```
Accuracy = (TP + TN)/Total

Precision = TP/(TP + FP)

Recall = TP/(TP + FN)

F1 score = 2*((Precision*Recall)/(Precision+Recall))
```
#### Which metrics matter most?
For our breast cancer dataset, predicting ALL malignant cases as malignant is of the utmost importance -- and even if there are some false positives (benign cases that are marked as malignant), these likely will be discovered by follow-up tests.  Whereas missing a malignant case (classifying it as benign) could have deadly consequences.  Thus, we want to minimize false negatives. This in turn will maximize the recall ratio (also known as the sensitivity or true positive rate). 

`scikit-learn` Implementation

Logistic regression not only predicts the class of a sample, but also the probability of a sample belonging to each class. It provides us with a measure of certainty associated with each prediction. In the default implementation in `scikit-learn`, a probability greater than 50% means that the predicted outcome will belong to the positive class. This is referred to as a prediction threshold. If two samples have predicted probabilities of 51% and 99%, both will be considered positive with the default threshold.  However, if the threshold is increased to 60%, a predicted probability of 51% will be assigned the negative class.

![thresholding](https://static-assets.codecademy.com/Paths/machine-learning-engineer-career-path/logistic-regression/thresholding.png)

Consider the histogram of the predicted probabilities for the logistic regression classifier shown above.  The benign (or negative class) is depicted in blue, and the malignant (or positive class) in orange for the breast cancer data set.  The benign cases are heavily clustered around zero, which is good as they will be correctly classified as benign, whereas malignant cases are heavily clustered around one.  The vertical lines depict hypothetical threshold values at 25%, 50%, and 75%.  For the highest threshold, almost all the samples above 75% belong to the malignant class, but there will be some benign cases that are misdiagnosed as malignant (false positives).  In addition, there are a number of malignant cases that are missed (false negatives).  If instead the lowest threshold value is used, almost all the malignant cases are identified, but there are more false positives.  

Therefore, the value of the threshold is an additional lever that can be used to tune a model's predictions. A higher value is generally associated with fewer false positives and more false negatives. Whereas a lower value is associated with fewer false negatives and more false positives.

Prediction Thresholds

Nice work! Let's review all the concepts you've learned in this lesson:

- The logistic regression primary assumptions include: 
    1. The target variable is binary
    2. The observations are independent of one another
    3. The sample size must be large enough
    4. There should not be any extreme outliers in the data

- Additional assumptions of LR include: 
    1. Features are linearly related to the logit of the outcome (sigmoidal curve)
    2. No multicollinearity (can cause the coefficients and p-values to be inaccurate)

- We can train and predict a model using `scikit-learn`. 

- Hyperparameters are set before the model implementation step and tuned later to improve model performance.

- We can use metrics (accuracy, precision, recall, or F1-score) to evaluate our Logistic Regression model.

- The prediction threshold (a measure of certainty associated with each prediction) can be an additional lever to tune a model’s predictions. In our breast cancer data, we wanted a lower threshold with fewer false negatives (a malignant tumor classified as benign).

- For each prediction threshold, the True Positive Rate (TPR) and the False Positive Rate (FPR) can be calculated and then plotted. The resulting curve these points form is known as the Receiver Operating Characteristic (ROC) curve.

- The AUC (Area Under the Curve) tells us how well our model can distinguish between the two classes. An AUC score close to 1 is a near-perfect classifier, whereas a value of 0.5 is equivalent to random guessing.

- If your classes are significantly imbalanced, this could create a bias towards the majority class since the model learns that it can have a higher accuracy if it predicts the majority class more often (not good).

- To bring the positivity rate of the dataset closer to .5, we can undersample the majority class, oversample the minority class, or balance the weights with `class_weight='balanced'`.