pyspark AttributeError: 'DataFrame' object has no attribute 'toDF'

Question

The following code worked for me before, but not anymore. I got the error:

AttributeError: 'DataFrame' object has no attribute 'toDF'

if __name__ == "__main__":
  sc = SparkContext(appName="test")
  sqlContext = SQLContext(sc)

  df = sqlContext.read.format('com.databricks.spark.csv').\
    options(header='false',delimiter=',',inferSchema='true').load('test')

  ### rename columns
  df = df.toDF('a','b','c')
  ...
  sc.stop()

Possible duplicate of [How to change dataframe column names in pyspark?](http://stackoverflow.com/questions/34077353/how-to-change-dataframe-column-names-in-pyspark) — David, Jul 26 '16 at 17:30
I am aware of that post. I am just thinking 'toDF' is more convenient and it worked for me before. https://spark.apache.org/docs/1.6.1/api/python/pyspark.sql.html — user3610141, Jul 26 '16 at 21:02
I figured it out. Looks like it has to do with our spark version. It worked with 1.6. — user3610141, Jul 27 '16 at 20:32

score 1 · Accepted Answer · answered Jul 27 '16 at 20:35

1

I figured it out. Looks like it has to do with our spark version. It worked with 1.6

answered Jul 27 '16 at 20:35

user3610141

165
2
12

score 0 · Answer 2 · answered Dec 06 '18 at 20:49

if you are working with spark version 1.6 then use this code for conversion of rdd into df

from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame(rdd)

if you want to assign title to rows then use this

df= rdd.map(lambda p: Row(ip=p[0], time=p[1], zone=p[2]))

ip,time,zone are row headers in this example.

pyspark AttributeError: 'DataFrame' object has no attribute 'toDF'

2 Answers2