3 Methods to Create Conditional Columns with Python Pandas and Numpy

Python is arguably the most popular programming language in the data science ecosystem. One of the reasons for such a popularity is the rich selection of Python libraries for data science.

These libraries offer fast, flexible, and efficient methods for performing data operations. Pandas and Numpy are two popular Python libraries used for data analysis and manipulation tasks.

Deriving new columns based on the existing ones in a dataset is a typical task in data preprocessing. It is an essential part of feature engineering as well. In some cases, the new columns are created according to some conditions on the other columns.

In this article, we will go over 3 methods that can be used for creating a new column based on some conditions on the other columns. Pandas and Numpy offer great flexibility for performing this task as we will see in the examples.

Let's start with importing libraries and creating a sample data frame. The following data frame contains a part from the Melbourne housing dataset on Kaggle.

import numpy as npimport pandas as pd

melb = pd.read_csv("/content/melb_data.csv", usecols = ["Address", "Regionname", "Type", "Rooms", "Distance", "Price"])

melb.head()

The dataset contains features about some houses in Melbourne along with their prices.

1. Pandas where function

The first method is the where function of Pandas. It allows for creating a new column according to the following rules or criteria:

• The values that fit the condition remain the same

• The values that do not fit the condition are replaced with the given value

As an example, we can create a new column based on the price column. If the price is higher than 1.4 million, the new column takes the value "class1". Otherwise, it takes the same value as in the price column.

melb["new1"] = melb.Price.where(melb.Price < 1400000, "class1")

melb.head()

The first parameter of the where function specifies the condition. The second one is the value to replace values that do not fit the condition. The condition in our case is the price being less than 1.4 million. The first, third, and fifth columns do not fit the condition so the new column takes the "class1" value in these rows.

2. Numpy where function

Although they have the same name, the where function of Pandas and Numpy are very different. First of all, the where function of Numpy provides greater flexibility. How it treats the given condition is also different from Pandas.

Pandas where function only allows for updating the values that do not meet the given condition. However, the where function of Numpy allows for updating values that meet and do not meet the given condition.

The following code block creates a column named "new2". The first parameter in the where function specifies the condition. The second parameter indicates the value to be used for rows that fit the condition. The third parameter is the value for the rows that do not meet the given condition.

melb["new2"] = np.where(   (melb.Rooms == 3) & (melb.Price > 1400000),    "class1",    "other")

melb.head()

Only the third row in the screenshot fits the given set of conditions so it takes the value "class1".

3. Numpy select function

The select function is more capable than the previous two methods. We can use it to give a set of conditions and a set of values. Thus, we are able to assign a specific value for each condition.

Let's first define the conditions and associated values.

filters = [   (melb.Rooms == 3) &amp; (melb.Price > 1400000),   (melb.Rooms == 2) &amp; (melb.Price < 1400000),   (melb.Price < 900000)]

values = ["class1", "class2", "class3"]

If a house has 3 rooms and its price is more than 1.4 million, the new column takes the value "class1". If a house has 2 rooms and its price is less than 1.4 million, the value is "class2", and so on. It is similar to writing a bunch of if-else statements.

We can apply these conditions and values with the select function as follows:

melb["new3"] = np.select(filters, values)

melb.head()

The first and fifth rows do not meet any of the given conditions. By default, these rows take the value 0. However, it can be changed using the default parameter.

Here is another example that demonstrates the use of the default parameter.

melb["new4"] = np.select(filters, values, default="no_class")

melb.head()

Conclusion

We have covered 3 different methods for creating new columns in a data frame conditionally. Depending on the flexibility you need or the complexity of the task, you can choose the best method for you.

Always keep in mind that the simpler is the better. Thus, go for the simple method unless you need to use multiple conditions and values.

Thank you for reading. Please let me know if you have any feedback.