How to use Pandas concat() to concatenate dataframes

The Pandas concat() function is used to concatenate (or join together) two or more Pandas objects such as dataframes or series. It can be used to join two dataframes together vertically or horizontally, or add additional rows or columns.

Pandas concat() is an important function to learn, since the function usually used for these tasks - the Pandas append() function - is deprecated since version 1.4.0 and may eventually be removed from Pandas.

This is mildly annoying, as I prefer append(), but concat() has the benefit of being a bit quicker as it adds additional data in a single operation, rather than doing it iteratively. That’s going to have a more noticeable effect on larger datasets than those I work on. However, while it’s a bit fiddlier to use, it’s faster and the recommended way to concatenate data, so you need to learn it.

The concat() function

The concat() function has various optional parameters, but the essential one you need to pass is called objs and defines the Pandas objects you want to concatenate - whether that’s Series or DataFrame objects. There are also a number of other optional parameters. Here’s a quick summary of what they do and why you might need to use them.

Import Pandas

To get started, open a Jupyter notebook and import the Pandas library using the as pd naming convention.

import pandas as pd

Use concat() to concatenate two dataframes vertically

First, we’ll use concat() to concatenate or join together two dataframes vertically. We’ll create two dataframes that contain the same column names.

df1 = pd.DataFrame(
    [('Pterophyllum altum', 3, 12.5, 13.3), 
     ('Pterophyllum scalare', 2, 10.0, 11.0),
     ('Pterophyllum leopoldi', 1, 8.0, 9.0)], 
    columns=['species', 'age', 'length', 'weight']
)
df1

df2 = pd.DataFrame(
    [('Vieja synspila', 2, 10.0, 11.0),
    ('Altolamprologus calvus', 1, 8.0, 9.0)],
    columns=['species', 'age', 'length', 'weight']
)
df2

To concatenate the dataframes vertically (i.e. one below the other), we can use the concat function with no additional arguments. We’ll pass the objs parameter a list of the dataframes we want to concatenate. Since the column names are the same, we end up with a neat dataframe with df1 stacked vertically on top of df2 beneath.

df3 = pd.concat([df1, df2])
df3

Use concat() with ignore_index=True

If you look closely at the example above, you’ll spot that the index doesn’t look quite right, since it contains several 0 and 1 values from concatenating the two dataframes. To fix this, you can use the ignore_index parameter to create a new index. Now we get back a neater dataframe in which the index is reset to appear in the correct order.

df4 = pd.concat([df1, df2], ignore_index=True)
df4

Use concat() to concatenate a new row

The concat() function can also be used to add a new row to a DataFrame. There are a couple of ways to do this. The first way is to create a new dataframe with the same columns and index with a single row, and then concatenate it vertically using the ignore_index=True argument.

new_row = pd.DataFrame(
    [('Pterophyllum altum', 4, 14.0, 15.0)],
    columns=['species', 'age', 'length', 'weight']
)

df5 = pd.concat([df4, new_row], ignore_index=True)
df5

Use concat() to add a new column to a dataframe

The concat() method can also be used to add a new column to a dataframe. To do this, we need to create a new series using pd.Series() and then concatenate it to the other dataframe. When calling pd.Series() you need to ensure you provide the correct number of values, and define the name parameter to give the new column a name.

df6 = pd.concat([df5, pd.Series([5, 9, 23, 4, 9, 23], name='price')], axis=1)
df6

Matt Clarke, Sunday, November 27, 2022