Preparing Your Data for Machine Learning Models: Dealing with Missing Values using Scikit Learn

Handling missing values in your data is a critical step in preparing your data for machine learning models. Missing values can introduce bias and reduce the effectiveness of your models if not handled properly. In this tutorial, we will explore how to handle missing values using the Scikit-learn library in Python.

Step 1: Import the necessary libraries

Before we can begin handling missing values, we need to import the necessary libraries. We will be using pandas for data manipulation and Scikit-learn for machine learning models.

import pandas as pd
from sklearn.impute import SimpleImputer

Step 2: Load your data

Next, we need to load the data that contains missing values. For this tutorial, let’s use a sample dataset from sklearn.datasets.

from sklearn.datasets import load_iris
data = load_iris()
df = pd.DataFrame(data.data, columns=data.feature_names)

Step 3: Inspect the data

Before we start handling missing values, it’s important to first understand the extent of missing values in your dataset. You can use the isnull() method to check for missing values.

print(df.isnull().sum())

This will show you the number of missing values for each column in your dataset.

Step 4: Handle missing values with SimpleImputer

The SimpleImputer class from Scikit-learn provides a simple strategy to handle missing values. You can specify the strategy (mean, median, most_frequent, or constant) to impute the missing values.

imputer = SimpleImputer(strategy='mean')
imputer.fit(df)
df_imputed = imputer.transform(df)

Step 5: Convert the imputed array back to a DataFrame

After imputing the missing values, we need to convert the imputed array back to a DataFrame for further analysis or modeling.

df_imputed = pd.DataFrame(df_imputed, columns=df.columns)

Step 6: Verify missing values have been handled

To ensure that missing values have been successfully handled, you can check for missing values in the imputed DataFrame.

print(df_imputed.isnull().sum())

If there are no missing values, then you have successfully handled missing values in your dataset.

Step 7: Use the imputed data for machine learning models

Now that missing values have been handled, you can use the imputed data for training machine learning models. You can proceed with splitting the data into training and test sets, and training the model using Scikit-learn.

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(df_imputed, data.target, test_size=0.2, random_state=42)
clf = RandomForestClassifier()
clf.fit(X_train, y_train)

By following these steps, you can effectively handle missing values in your data using Scikit-learn. Handling missing values is a crucial step in preparing your data for machine learning models, and using the SimpleImputer class makes this process simple and straightforward.