How to use method chaining in Pandas

Pandas method chaining, or flow programming, is a modern, but sometimes controversial way of structuring Pandas code into a structured chain or series of commands. Conceptually, Pandas chaining is a bit like the scikit-learn pipeline methodology used in machine learning.

Chaining is present in other data science libraries, such as R’s dplyr, but Pandas lends itself to chaining well because so many Pandas functions return a dataframe as their output. Each operation in the chain performs an action on the dataframe and passes it to the next step, allowing you to create a step-by-step series of operations that output a final dataframe while simultaneously avoiding the creation of temporary intermediate variables that bloat your code and increase memory usage.

I’m a huge fan of Python author Matt Harrison and he is a big advocate of the technique and I’ve learned a lot from his writing on this topic. As Matt says, it can take a little getting used to, and at first may seem harder to understand, but many data scientists see it as a game changer in terms of helping them create more readable sequential code after they adopt the technique.

In this tutorial, I’ll explain the advantages and disadvantages of method chaining, and show some simple code examples that convert regular Pandas code to code written using method chaining.

Advantages of method chaining in Pandas

Method chaining advocates say that it makes code clearer to read, quicker to write, avoids errors, and improves performance. That said, this does depend on the length and complexity of the chain you’re creating. It can be useful, but it can also be complex to decipher.

Disadvantages of method chaining in Pandas

Method chaining is a fairly divisive technique and some people love it while others hate it. I have similar feelings about Object Relational Mapping or ORM in SQL. Both methods are designed to make code quicker and clearer to write, but share similar problems in that they can become complex and can sometimes make debugging much harder.

Splitting up larger chains into smaller steps can help resolve these issues, and there are some neat debugging methods you can employ that overcome many of the criticisms of the technique, so the issues are rarely insurmountable.

Import the packages

To get started, open a Jupyter notebook and import the Pandas library.

import pandas as pd

Write some code without method chaining

To show how method chaining works, we’ll first write some regular code in Jupyter cells and then refactor it to use method chaining. This will show how method chaining can make code more readable and concise. Ignore the fact that we’re performing some pointless calculations - it’s just for demonstration purposes.

df = pd.read_csv('https://raw.githubusercontent.com/flyandlure/datasets/master/ecommerce_sales_by_date.csv')
df = df.fillna('')
df = df.sort_values(by='date', ascending=False)
df['conversion_rate'] = ((df['transactions'] / df['sessions']) * 100).round(2)
df['revenuePerTransaction'] = (df['transactionRevenue'] / df['transactions']).round(2)
df['transactionsPerSession'] = (df['transactions'] / df['sessions']).round(2)
df['date'] = pd.to_datetime(df['date'])
df = df.rename(columns={'date': 'Date', 
                        'sessions': 'Sessions', 
                        'transactions': 'Transactions', 
                        'transactionRevenue': 'Revenue', 
                        'transactionsPerSession': 'Transactions Per Session', 
                        'revenuePerTransaction': 'AOV',
                        'conversion_rate': 'CR'}) 
df = df.drop(columns=['Unnamed: 0', 'Transactions Per Session'])
df.head()

Refactor the code to use method chaining

Next we’ll refactor the code above to use method chaining. Method chaining works by using a pair of parentheses, which allow you to include whitespace or linebreaks in your code to aid readability. We’re effectively creating a one-liner that performs all steps, but we’re using the parentheses to allow us to split it over multiple lines to make it readable and logical.

We’ll add each part of the chain using a step-by-step process. First, we’ll call pd.read_csv() and load our remote CSV file. This returns a raw dataframe that gets passed to the next step, which uses fillna() to fill any missing values.

The fillna() step returns a dataframe which then gets passed to the next step, which uses assign() to run some lambda functions that perform some calculations and use round() to round the values. We’ll then use rename() to rename some columns, use drop() to drop some columns, and use astype() to cast the dtypes of the date column.

df_chain = (
    pd.read_csv('https://raw.githubusercontent.com/flyandlure/datasets/master/ecommerce_sales_by_date.csv')
    .fillna('')
    .sort_values(by='date', ascending=False)
    .assign(
        conversion_rate=(lambda x: ((x['transactions'] / x['sessions']) * 100).round(2)),
        revenuePerTransaction=(lambda x: x['revenuePerTransaction'].round(2)),
        transactionsPerSession=(lambda x: x['transactionsPerSession'].round(2))
    )
    .rename(columns={'date': 'Date', 
                    'sessions': 'Sessions', 
                    'transactions': 'Transactions', 
                    'transactionRevenue': 'Revenue', 
                    'transactionsPerSession': 'Transactions Per Session', 
                    'revenuePerTransaction': 'AOV',
                    'conversion_rate': 'CR'})        
    .drop(columns=['Unnamed: 0', 'Transactions Per Session'])
    .astype({'Date': 'datetime64[ns]'})  
)

df_chain.head()

The code does exactly the same thing but it is much more readable and easier to understand. In the earlier code, since it could be split up over multiple cells in a Jupyter notebook, there’s always a risk that someone might run the code out of sequence and end up with a different result. However, since method chaining groups everything in a single line, this risk is completely eliminated.

Functions that can be used in method chaining

Method chaining supports any Pandas function that takes a dataframe as its input, modifies the dataframe, and returns a dataframe as its output so it can be passed to the next step in the chain. Here’s a summary of some of the most commonly used Pandas functions in method chaining.

Further reading

• SDS 557: Effective Pandas — with Matt Harrison, Super Data Science podcast with Jon Krohn.
• PyParis 2018 - Modern Pandas - Writing effective, readable data pipeline, Pôle Systematic Paris Region.

Matt Clarke, Thursday, January 05, 2023