How to use the Pandas filter() function

The Pandas filter() function is used to filter a dataframe based on the column names, rather than the column values, and is useful in creating a subset dataframe containing only those columns in which you are interested.

While the Pandas query() function might look similar, query() filters a dataframe based on the column values, not the column names, so it’s quite different really.

In this tutorial, we’ll go over the basic features of the Pandas filter() method so you can see how easy it is to use. Since filter() takes a dataframe as its input and returns a transformed dataframe as its output, it can be used with the modern method chaining technique.

Using the Pandas filter() method

The Pandas filter() method is very simple. It’s just another way of sub-setting a dataframe to include only those columns whose names match your search criteria. It takes four arguments:

Create a Pandas dataframe

To get started working through these simple examples, open a Jupyter notebook and import the Pandas library using the import pandas as pd naming convention. Then, either import your own data to filter, or create a Pandas dataframe containing some dummy data.

import pandas as pd

df = pd.DataFrame(
    [('Pterophyllum altum', 'Cichlidae', 15, 12.3),
        ('Pterophyllum scalare', 'Cichlidae', 10, 9.2),
        ('Pterophyllum leopoldi', 'Cichlidae', 8, 6.5), 
        ('Corydoras paleatus', 'Callichthyidae', 5, 0.5),
        ('Corydoras aeneus', 'Callichthyidae', 7, 2.5),
        ('Corydoras trilineatus', 'Callichthyidae', 5, 1.5),
        ('Corydoras habrosus', 'Callichthyidae', 3, 0.5),
    ],
    columns=['species', 'family', 'length', 'weight']
)
df

Use filter() to select specific columns

At its most basic, the filter() method takes an argument called items which accepts a list of Pandas column names you want to select. Calling df.filter(items=['species', 'family']) does exactly the same as df[['species', 'family']].

When you call filter() it returns a transformed dataframe and doesn’t change the original, so if you want to retain the transformed dataframe you’ll need to save it to a new variable or overwrite the original. For example, df_subset = df.filter(items=['species', 'family']).

df.filter(items=['species', 'family'])

Use filter() to select column names containing a string

The like argument is used to select only those columns that contain a specific value. For example, to return a subset dataframe containing only the column names that contain an e we can pass in the argument like='e' and set axis=1.

df.filter(like='e', axis=1)

Use filter() to select column names based on a regex

The like argument returns column names that match a substring value. However, the regex argument allows you to pass in a Python regular expression. For example, to return only the columns that start with len we can use the regex ^len.

df.filter(regex='^len', axis=1)

Use the filter() function with method chaining

As mentioned above, since the filter() function is appended to a Pandas dataframe object to obtain its input data and returns a dataframe as its output, it lends itself to being used in method chaining. Each step in the chain transforms the dataframe and passes its output to the next step, creating a list of operations that are performed sequentially.

df_subset = (
    df
    .filter(items=['species', 'family', 'weight'])
    .query('family == "Cichlidae"')
)

df_subset

Matt Clarke, Monday, January 16, 2023