Learn about how to build machine learning pipelines using `scikit-learn`!

Machine Learning Pipelines

Often times, you may not want to simply apply every function to all columns.  If our columns are of different types, we may only want to apply certain parts of the pipeline to a subset of columns.  This is what we saw in the two previous exercises.  One set of transformations are applied to numeric columns and another set to the categorical ones.  We can use `scikit-learn`'s `ColumnTransformer` as one way of combining these processes together.

`ColumnTransformer` takes in a list of tuples of the form `(name, pipeline, columns)`:
```
example_column_transformer = ColumnTransformer(
    transformers=[ ("name_1", pipeline_1, columns_1),
                   ("name_2", pipeline_2, columns_2)])
``` 


The transformer can be anything with a `.fit` and `.transform` method like we used previously (like `SimpleImputer` or `StandardScaler`), but can also itself be a pipeline, as we will use in the exercise.  


Column Transformer

Way to go! Now that we are getting the hang of pipelines, we're going take things up a notch. We will now be searching over different types of models, each having their own sets of hyperparameters! In the original `pipeline`, we defined `regr` to be an instance of `LinearRegression()`. Then in defining the parameter grid to search over, we used the dictionary `{"regr__fit_intercept": [True,False]}` to define the values of the `fit_intercept` term. We can equivalently do this by passing both the estimator AND parameters in a single dictionary as
```
{'regr': [LinearRegression()], "regr__fit_intercept": [True,False]}
```
We can add more models to it as follows. Suppose we wanted to add a Ridge regression model and also perform hyperparamter tuning using `GridSearchCV` to find the best regularization parameter `alpha`, we would add it to previous dictionary to create an array of dictionaries as follows:
```
search_space = [{'regr': [LinearRegression()], 'regr__fit_intercept': [True,False]},
                {'regr':[Ridge()], 'regr__alpha': [0,0.1,1,10,100]}
```
_Note_: If you'd like a refresher on regularization using hyperparameter tuning in regression models, check out our [article](https://author.codecademy.com/content-items/1a6ba4626608600468467b86bdc03181) and [lesson](https://author.codecademy.com/content-items/b9ddf95b9ffd49cfb3d2c361cec4a7da) on the same.


The goal of this process is to find the best estimator for our dataset and problem in the most efficient manner possible. The pipeline module allows us to do exactly that! In a couple of lines of code, we're able to preprocess the data and search an entire model _and_ hyperparameter space. The final step is to access the pipeline elements to draw out the information about which estimator and hyperparameter set gets us the best score. We do this by using the `.next_steps` method by using the strings we've used in the dictionary. For instance, the regression model can be access using the string `'regr'` from the dictionary as follows:
- Get the best estimator using `GridSearchCV`'s `.best_estimator_` method
- Use `.named_steps['regr'].get_params()` on the best estimator to get its hyperparameters!


Final Pipeline

Great!  Now that we have all the preprocessing done and coded succinctly using `ColumnTransformer` and `Pipeline`, we can add a model.  We will take the result at the end of the previous exercise, and now create a final pipeline with the `ColumnTransformer` as the first step, and a `LinearRegression()` model as the second step.

By adding a model to the final step, the last step no longer has a `.transform` method.  This is the only step in a pipeline that can be a non-transformer.  But now the final step also has a `.predict` method, which can be called on the entire pipeline! Additionally the `.score()` method, which estimates the default prediction score on any `scikit-learn` model can also be used to evaluate the performance of the pipeline.

Adding a Model

To introduce pipelines, let's look at a common set of data cleaning/EDA tasks &mdash; dealing with missing values and scaling numeric variables. We're going to convert an existing code base that performs these tasks to more concise code that uses `scikit-learn`'s `Pipeline` using the following steps.

1. First, to define a pipeline, we pass a list of tuples of the form `(name, transform/estimator)` into a `Pipeline` object.  For example, if we wanted to perform imputation with a `SimpleImputer` first, and scale our numerical variables with a `StandardScaler` next, the code would look as follows:
```py
from sklearn.pipeline import Pipeline
pipeline = Pipeline([("imputer",SimpleImputer()), ("scale",StandardScaler())])
```
2. Once a `Pipeline` object has been instantiated, the methods `.fit` and `.transform` can be called like we would with any data transformation object in `scikit-learn`. 
```py
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X,y, random_state=0, test_size=0.25)
pipeline.fit(x_train)
pipeline.transform(x_test)
``` 

If the pipeline includes a machine learning model as well,`.predict` can also be called down the line. Each step in the pipeline will be fit in the order provided.  Further parameters can be passed to each step as well.  For example, if we want to pass the parameter `with_mean=False` to the `StandardScaler`, we'd use:
```py
Pipeline([("imputer",SimpleImputer()), ("scale",StandardScaler(with_mean=False))])
```

In the code editor, we've loaded a [dataset](https://archive.ics.uci.edu/ml/datasets/abalone) from the UCI Machine Learning repository containing information about different attributes of [abalones](https://en.wikipedia.org/wiki/Abalone), typically used to predict the age of the abalone. We've defined the predictor and target variables, identified the numerical and categorical columns, added some missing values to impute, and performed a train-test split.





Data Cleaning (Numeric)

While scikit-learn contains many existing transformers and classes that can be used in pipelines, you may need at some point to create your own.  This is simpler than you may think, as a step in the pipeline needs to have only a few methods implemented.  If it is an intermediate step, it will need fit and transform methods. We will implement all of this in the exercise below! 

Here are some of the major takeaways on building pipelines in `scikit-learn`:

- Pipelines help make concise, reproducible, code by combining steps of transformers and/or a final estimator.

- Intermediate steps of a pipeline must have both the `.fit()` and `.transform()` methods. This includes preprocessing, imputation, feature selection, dimension reduction. 

- The final step of a pipeline must have the `.fit()` method -- this can include a transformer or an estimator/model.

- If the pipeline is meant to only transform your data by combining preprocessing and data cleaning steps, then each step in the pipeline will be a transformer.  If your pipeline will also include a model (a final estimation or prediction step), then the last step must be an estimator.  

- Once the steps of a pipeline are defined, it can be used like an other transformer/estimator by calling fit, transform, and/or predict methods.  Similarly, it can be used in place of an estimator in a hyperparameter grid search.

Writing Custom Classes & Summary

Great, we have a very condensed bit of code that does all our data cleaning, preprocessing, and modeling in a reusable fashion!  What now?  Well, we can tune some of the parameters of the model by applying a grid search over a range of hyperparameter values.

A linear regression model has very few hyperparameters and here we'll be using the hyperparameter that pertains to whether we include an intercept or not.  As we've aseen, the pipeline created in the previous exercise is an estimator and we can call the `.fit()` and `.predict()` methods on it.  So in fact, the whole pipeline can be passed as an estimator for `GridSearchCV`.  This will then refit the pipeline for each combination of parameter values in the grid and each fold in the cross-validation split.  

That's a lot -- but the code is again very short.  One last thing to keep in mind while referencing hyperparameters in a pipeline is the following: any hyperparameter can be called using `pipeline_step_name + '__' + hyperparameter`.  For example, `regr__fit_intercept` corresponds to a pipeline step named "regr" and the hyperparameter "fit_intercept". Let's see how this all works, shall we?

_Note_: If you'd like a refresher on how hyperparameters can be tuned check out our [lesson](https://www.codecademy.com/content-items/b9ddf95b9ffd49cfb3d2c361cec4a7da/exercises/introduction) on the same.

Hyperparameter Tuning

We're now going to implement a task similar to the previous exercise with `pipeline.Pipeline()`, but with categorical variables now. Specifically, we'll be dealing with missing values in categorical data and one-hot-encoding categorical variables. We will convert an existing codebase to a pipeline like in the previous exercise. The two steps in detail are:

1. `SimpleImputer()` will be used again to fill missing values in the pipeline, but this time, the strategy parameter will need to be updated to `most_frequent`.
2. [`OneHotEncoder()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder) will be used as the second step in the pipeline. The default setting in `scikit-learn`'s `OneHotEncoder()` is that a sparse array will be returned from this transform, so we will use `sparse='False'` to return a full array.

Data Cleaning (Categorical)

In this lesson we're going to learn how to turn a machine learning (ML) workflow to a pipeline using `scikit-learn`. A ML pipeline is a modular sequence of objects that codifies and automates a ML workflow to make it efficient, reproducible and generalizable. While the process of building pipelines is not singular, there are some tools that are universally used to do this. The most accessible of these is `scikit-learn`'s `Pipeline` object which allows us to chain together the different steps that go into a ML workflow. 

Turning a workflow into a pipeline has many other advantages too. Pipelines provide consistency &mdash; the same steps will always be applied in the same order under the same conditions. They also are very concise and can streamline your code. The `Pipeline` object within `scikit-learn` has consistent methods to use the many other estimators and transformers we have already covered in our ML curriculum. It is usually the starting point for a Machine Learning Engineer before turning to more sophisticated tools for scaling pipelines (such as PySpark, etc) and we will delve deeper into it in this lesson

What can go into a pipeline?  For any of the intermediate steps, it must have both the `.fit`  and `.transform` methods.  This includes preprocessing, imputation, feature selection and dimensionality reduction.  The final step must have the `.fit` method.  Examples of tasks we've seen already that could benefit from a pipeline include:
* scaling data then applying principal component analysis
* filling in missing values then fitting a regression model
* one-hot-encoding categorical variables and scaling numerical variables

In the following exercises, we will walk through various chained functions and how to incorporate these into a `Pipeline`.