[Pyspark] pandas dataframe to pyspark dataframe

TIL/잡다한

재융 2019. 1. 18. 11:58

Sometimes we have to change pandas.dataframe to pyspark.dataframe

there one solution exist

1
2
3
4
5
from pyspark.sql import SQLContext
 
temp_csv = pd.read_csv("./tmep.csv")
 
temp_pyspark = sqlContext.createDataFrame(temp_csv)

20200304

가끔 위 방법이 안될때가 있어서 새로운 방법 추가

1
2
3
4
5
temp_csv = pd.read_csv("./tmep.csv")

temp_pyspark = spark.createDataFrame(temp_csv)