How to Delete Rows Based on Column Values in a DataFrame?

remove rows based on column value pandas pyspark

A row in a DataFrame can be considered as an observation with several features that are represented by columns. We sometimes need to remove observations whose feature values do not fit the given condition.ย In this how-to article, we will learn how to delete rows based on column values in Pandas and PySpark DataFrames.

How to Delete Rows Based on Column Values in a DataFrame?

Pandas Delete Rows with Undesired Condition

We usually have multiple ways of writing the desired or undesired condition.

				
					# not equals
df = df[df["Item"] != "Broccoli"]

# tilde (not) operator
df = df[~(df["Item"] == "Broccoli")]
				
			

The tilde operator is especially useful when the undesired condition consists of multiple values. Letโ€™s say we have a larger DataFrame and want to delete rows with Broccoli, Potato, and Cucumber. Here are different ways of doing this operation:

				
					# not equals and & operator
df = df[
           (df["Item"] != "Broccoli") & 
           (df["Item"] != "Cucumber") & 
           (df["Item"] != "Potato")
       ]

# tilde (not) operator and isin method
df[~(df["Item"].isin(["Broccoli","Cucumber","Potato"]))]

# isin method and False
df = df[df["Item"].isin(["Broccoli","Cucumber","Potato"]) == False]
				
			

As we see in the third method above, the False condition can be used instead of the tilde operator.

PySpark Delete Rows with Undesired Conditionย 

We can use the filter or where function. The syntax is quite similar to the syntax of Pandas.

				
					# filter function
df = df.filter(df["Item"] != "Broccoli")

# where function
df = df.where(df["Item"] != "Broccoli")
				
			

With multiple undesired conditions, we can use the isin method with tilde operator or False condition.

				
					# isin and tilde
df = df.filter(~df["Item"].isin(["Broccoli","Cucumber","Potato"]))

# isin and False
df = df.filter(df["Item"].isin(["Broccoli","Cucumber","Potato"]) == False)
				
			

This question was also being asked as:

  • How to delete rows from a Pandas DataFrame based on a conditional expression
  • Delete columns and rows containing specific values in Pandas DataFrame
  • Remove rows from DataFrame based on condition in PySpark

People have also asked for:

You may also like

Start Monitoring Your Models in Minutes