[Pyspark] pyspark 함수 정리(2)

drop(*cols)

dataframe에서 지우고싶은 column이 있을때 해당 함수를 사용한다. 반환되는 값은 삭제가된 이후의 dataframe이 나오게 된다

>>> df.drop('age').collect()
[Row(name='Alice'), Row(name='Bob')]
 
>>> df.drop(df.age).collect()
[Row(name='Alice'), Row(name='Bob')]
 
>>> df.join(df2, df.name == df2.name, 'inner').drop(df.name).collect()
[Row(age=5, height=85, name='Bob')]
 
>>> df.join(df2, df.name == df2.name, 'inner').drop(df2.name).collect()
[Row(age=5, name='Bob', height=85)]
 
>>> df.join(df2, 'name', 'inner').drop('age', 'height').collect()
[Row(name='Bob')]

dropDuplicates 또는 drop_duplicates

dataframe에 중복된 row가 있을경우 중복된것을 삭제해준다

>>> from pyspark.sql import Row
>>> df = sc.parallelize([ \
...     Row(name='Alice', age=5, height=80), \
...     Row(name='Alice', age=5, height=80), \
...     Row(name='Alice', age=10, height=80)]).toDF()
>>> df.dropDuplicates().show()
+---+------+-----+
|age|height| name|
+---+------+-----+
|  5|    80|Alice|
| 10|    80|Alice|
+---+------+-----+
>>> df.dropDuplicates(['name', 'height']).show()
+---+------+-----+
|age|height| name|
+---+------+-----+
|  5|    80|Alice|
+---+------+-----+

첫번째에는 height과 name이 중복되지만 age가 중복이 안되서 삭제가 안된것이고, 두번째 dropDuplicates는 column을 지정해줘서 중복된값이 삭제된것을 볼 수가 있음

dropna(how='any', thresh=None, subset=None)

null값이 포함된 row를 뺀 새로운 dataframe을 반환해준다

Parameters:	how – ‘any’ or ‘all’. If ‘any’, drop a row if it contains any nulls. If ‘all’, drop a row only if all its values are null. thresh – int, default None If specified, drop rows that have less than thresh non-null values. This overwrites the howparameter. subset – optional list of column names to consider.

>>> df4.na.drop().show()
+---+------+-----+
|age|height| name|
+---+------+-----+
| 10|    80|Alice|
+---+------+-----+

dtypes

dataframe의 column별 데이터 타입을 반환해준다

>>> df.dtypes
[('age', 'int'), ('name', 'string')]

printSchema()

dtypes와 비슷하지만 트리형태로 column별 데이터 타입을 반환해줌

>>> df.printSchema()
root
 |-- age: integer (nullable = true)
 |-- name: string (nullable = true)
<BLANKLINE>

orderBy

지정된 column의 정렬된 DataFrame을 반환

사용법은 여러가지가 있는데 sort()라는 함수도 쓸수가 있다

>>> df.sort(df.age.desc()).collect()
[Row(age=5, name='Bob'), Row(age=2, name='Alice')]
>>> df.sort("age", ascending=False).collect()
[Row(age=5, name='Bob'), Row(age=2, name='Alice')]
>>> df.orderBy(df.age.desc()).collect()
[Row(age=5, name='Bob'), Row(age=2, name='Alice')]
>>> from pyspark.sql.functions import *
>>> df.sort(asc("age")).collect()
[Row(age=2, name='Alice'), Row(age=5, name='Bob')]
>>> df.orderBy(desc("age"), "name").collect()
[Row(age=5, name='Bob'), Row(age=2, name='Alice')]
>>> df.orderBy(["age", "name"], ascending=[0, 1]).collect()
[Row(age=5, name='Bob'), Row(age=2, name='Alice')]

replace

Dataframe.replace(to_replace, value, subset=None)

형식으로 되어있으며 예시는 다음과 같음

>>> df4.na.replace(10, 20).show()
+----+------+-----+
| age|height| name|
+----+------+-----+
|  20|    80|Alice|
|   5|  null|  Bob|
|null|  null|  Tom|
|null|  null| null|
+----+------+-----+
>>> df4.na.replace('Alice', None).show()
+----+------+----+
| age|height|name|
+----+------+----+
|  10|    80|null|
|   5|  null| Bob|
|null|  null| Tom|
|null|  null|null|
+----+------+----+
>>> df4.na.replace({'Alice': None}).show()
+----+------+----+
| age|height|name|
+----+------+----+
|  10|    80|null|
|   5|  null| Bob|
|null|  null| Tom|
|null|  null|null|
+----+------+----+
>>> df4.na.replace(['Alice', 'Bob'], ['A', 'B'], 'name').show()
+----+------+----+
| age|height|name|
+----+------+----+
|  10|    80|   A|
|   5|  null|   B|
|null|  null| Tom|
|null|  null|null|
+----+------+----+