DataFrame is a two-dimensional data structure, which consists of labeled rows and columns. Each row can be considered as a data point or observation, and the columns represent the features or attributes of the data points.
We can create a DataFrame by stacking data points (i.e. appending one row at a time). In this short how-to article, we will learn how to perform this task for Pandas and PySpark DataFrames.
We can use the loc method to append rows to a DataFrame. The number of values we provide must match the number of columns in the DataFrame.
Consider we have a list of names and a list of scores. We want to create a DataFrame from the values in these lists. It will be a DataFrame with two columns: Names and Score. The first step is to create an empty DataFrame with two columns. Then, we append rows in a for loop.
# import Pandas import pandas as pd # Python lists names = ["John", "Jane", "Abby", "Matt"] scores = [80, 94, 87, 85] # Create empty DataFrame df = pd.DataFrame(columns=["Name", "Score"], dtype="int") # Append rows for i in range(len(names)): df.loc[i] = [names[i], scores[i]]
The for loop takes the first items in both lists and appends as the first row to the DataFrame, and then the second ones, and so on.
We can add new rows to a DataFrame by using the union function, which is also used for combining DataFrames with the same schema. Before a row is appended to the DataFrame, it needs to be converted to a DataFrame. Therefore, this operation is more like appending a DataFrame with one row.
# import libraries and initialize a spark session from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringType, IntegerType spark = SparkSession.builder.getOrCreate() # Python lists names = ["John", "Jane", "Abby", "Matt"] scores = [80, 94, 87, 85] # Create empty DataFrame schema = StructType([ StructField("Names", StringType()), StructField("Scores", IntegerType()) ]) df = spark.createDataFrame(, schema) # Append rows for i in range(len(names)): row_to_append = [(names[i], scores[i])] columns = ["Names", "Scores"] df_to_append = spark.createDataFrame(row_to_append, columns) df = df.union(df_to_append)
This question is also being asked as:
- How to add an extra row to a Pandas DataFrame
- Add new rows to PySpark DataFrame