How to use isna() to check for missing values in a Pandas dataframe

Real world data is rarely clean, and you’ll often encounter missing values when working with Pandas dataframes. Missing values can lead to errors in your code, and can cause models to fail or be inaccurate, so it’s vital that you check for them.

In this project, I’ll show you how you can use the Pandas isna() function to check for missing values in your dataframe, and in specific columns, and how to create a helper function that determines the percentage of missing values overall, so you can decide on the best approach to dealing with them.

Import the packages

To get started, open a Jupyter notebook and import the Pandas package. You’ll likely already have Pandas installed, but if you don’t you can install it by entering !pip3 install pandas in a Jupyter cell or pip3 install pandas in your terminal.

import pandas as pd

Load the data

Next, we’ll load a dataset into Pandas that includes some missing values. I’ve used a dataset on food consumption from OpenMV, but you can use any data you like.

If you run df.info() on the dataframe you’ll see that it contains an object column containing the country name, and then a load of numeric data columns containing information on food consumption.

df = pd.read_csv('food-consumption.csv')
df.head()

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 16 entries, 0 to 15
Data columns (total 21 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   Country         16 non-null     object 
 1   Real coffee     16 non-null     int64  
 2   Instant coffee  16 non-null     int64  
 3   Tea             16 non-null     int64  
 4   Sweetener       15 non-null     float64
 5   Biscuits        15 non-null     float64
 6   Powder soup     16 non-null     int64  
 7   Tin soup        16 non-null     int64  
 8   Potatoes        16 non-null     int64  
 9   Frozen fish     16 non-null     int64  
 10  Frozen veggies  16 non-null     int64  
 11  Apples          16 non-null     int64  
 12  Oranges         16 non-null     int64  
 13  Tinned fruit    16 non-null     int64  
 14  Jam             16 non-null     int64  
 15  Garlic          16 non-null     int64  
 16  Butter          16 non-null     int64  
 17  Margarine       16 non-null     int64  
 18  Olive oil       16 non-null     int64  
 19  Yoghurt         15 non-null     float64
 20  Crisp bread     16 non-null     int64  
dtypes: float64(3), int64(17), object(1)
memory usage: 2.8+ KB

Check for missing values

The easiest way to check for missing values in a Pandas dataframe is via the isna() function. The isna() function returns a boolean (True or False) value if the Pandas column value is missing, so if you run df.isna() you’ll get back a dataframe showing you a load of boolean values.

df.isna().head()

That’s not usually very useful, so instead we’ll calculate the sum() of missing values by running df.isna().sum(). This returns the columns in our Pandas dataframe along with the number of missing values detected in each one, so 0 means there are no missing values, and 1 means there is a single missing value.

df.isna().sum()

Country           0
Real coffee       0
Instant coffee    0
Tea               0
Sweetener         1
Biscuits          1
Powder soup       0
Tin soup          0
Potatoes          0
Frozen fish       0
Frozen veggies    0
Apples            0
Oranges           0
Tinned fruit      0
Jam               0
Garlic            0
Butter            0
Margarine         0
Olive oil         0
Yoghurt           1
Crisp bread       0
dtype: int64

You can also run this code on a single Pandas column, or series, or a subset of columns, if you don’t need to check the entire dataframe.

df['Sweetener'].isna().sum()

1

df[['Sweetener', 'Frozen fish']].isna().sum()

Sweetener      1
Frozen fish    0
dtype: int64

Create a helper function to summarise missing data

To take this a step further, we can create a reusable helper function you can use in your projects. We’ll call this show_missing() and we’ll loop through each column in the dataframe and return some useful data on each one.

We’ll detect the Pandas dtypes, count the number of values, count the number of unique values, count the number of missing values, and calculate the percentage of values that are missing overall.

def show_missing(df):
    """Return a Pandas dataframe describing the contents of a source dataframe including missing values."""
    
    variables = []
    dtypes = []
    count = []
    unique = []
    missing = []
    pc_missing = []
    
    for item in df.columns:
        variables.append(item)
        dtypes.append(df[item].dtype)
        count.append(len(df[item]))
        unique.append(len(df[item].unique()))
        missing.append(df[item].isna().sum())
        pc_missing.append(round((df[item].isna().sum() / len(df[item])) * 100, 2))

    output = pd.DataFrame({
        'variable': variables, 
        'dtype': dtypes,
        'count': count,
        'unique': unique,
        'missing': missing, 
        'pc_missing': pc_missing
    })    
        
    return output

show_missing(df)

Dealing with missing values is a subject in its own right. There are various approaches you can use which range from imputing missing values to dropping them entirely.

Matt Clarke, Friday, September 02, 2022