Pandas – Filling NaN in Categorical data



Pandas – Filling NaN in Categorical data

Real-world data is full of missing values. In order to work on them, we need to impute these missing values and draw meaningful conclusions from them. In this article, we will discuss how to fill NaN values in Categorical Data. In the case of categorical features, we cannot use statistical imputation methods.

Let’s first create a sample dataset to understand methods of filling missing values:

# import modules
import numpy as np
import pandas as pd
# create dataset
data = {'Id': [1, 2, 3, 4, 5, 6, 7, 8],
        
        'Gender': ['M', 'M', 'F', np.nan,
                   np.nan, 'F', 'M', 'F'],
        
        'Color': [np.nan, "Red", "Blue",
                  "Red", np.nan, "Red",
                  "Green", np.nan]}
# convert to data frame
df = pd.DataFrame(data)
display(df)

Output:

 

To fill missing values in Categorical features, we can follow either of the approaches mentioned below –

Method 1: Filling with most occurring class

One approach to fill these missing values can be to replace them with the most common or occurring class. We can do this by taking the index of the most common class which can be determined by using value_counts() method. Let’s see the example of how it works:

# filling with most common class
df_clean = df.apply(lambda x: x.fillna(x.value_counts().index[0]))
df_clean

Output:

Method 2: Filling with unknown class

At times, the missing information is valuable itself, and to impute it with the most common class won’t be appropriate. In such a case, we can replace them with a value like “Unknown” or “Missing” using the fillna() method. Let’s look at an example of this –

 

# filling with Unknown class
df_clean = df.fillna("Unknown")
df_clean

Output:

Method 3: Using Categorical Imputer of sklearn-pandas library 

We have scikit learn imputer, but it works only for numerical data. So we have sklearn_pandas with the transformer equivalent to that, which can work with string data. It replaces missing values with the most frequent ones in that column. Let’s see an example of replacing NaN values of “Color” column – 

# using sklearn-pandas package
from sklearn_pandas import CategoricalImputer
# handling NaN values
imputer = CategoricalImputer()
data = np.array(df['Color'], dtype=object)
imputer.fit_transform(data)

Output:

 

Last Updated on October 23, 2021 by admin

Leave a Reply

Your email address will not be published. Required fields are marked *

Recommended Blogs