The most advanced ML Observability platform
We’re super excited to share that Aporia is now the first ML observability offering integration to the Databricks Lakehouse Platform. This partnership means that you can now effortlessly automate your data pipelines, monitor, visualize, and explain your ML models in production. Aporia and Databricks: A Match Made in Data Heaven One key benefit of this […]
Start integrating our products and tools.
We’re excited 😁 to share that Forbes has named Aporia a Next Billion-Dollar Company. This recognition comes on the heels of our recent $25 million Series A funding and is a huge testament that Aporia’s mission and the need for trust in AI are more relevant than ever. We are very proud to be listed […]
A row in a DataFrame can be considered as an observation with several features that are represented by columns. We sometimes need to remove observations whose feature values do not fit the given condition. In this how-to article, we will learn how to delete rows based on column values in Pandas and PySpark DataFrames.
We usually have multiple ways of writing the desired or undesired condition.
# not equals df = df[df["Item"] != "Broccoli"] # tilde (not) operator df = df[~(df["Item"] == "Broccoli")]
The tilde operator is especially useful when the undesired condition consists of multiple values. Let’s say we have a larger DataFrame and want to delete rows with Broccoli, Potato, and Cucumber. Here are different ways of doing this operation:
# not equals and & operator df = df[ (df["Item"] != "Broccoli") & (df["Item"] != "Cucumber") & (df["Item"] != "Potato") ] # tilde (not) operator and isin method df[~(df["Item"].isin(["Broccoli","Cucumber","Potato"]))] # isin method and False df = df[df["Item"].isin(["Broccoli","Cucumber","Potato"]) == False]
As we see in the third method above, the False condition can be used instead of the tilde operator.
We can use the filter or where function. The syntax is quite similar to the syntax of Pandas.
# filter function df = df.filter(df["Item"] != "Broccoli") # where function df = df.where(df["Item"] != "Broccoli")
With multiple undesired conditions, we can use the isin method with tilde operator or False condition.
# isin and tilde df = df.filter(~df["Item"].isin(["Broccoli","Cucumber","Potato"])) # isin and False df = df.filter(df["Item"].isin(["Broccoli","Cucumber","Potato"]) == False)