spark dataframe column string to date

Question

I would like to convert a spark dataframe string column of 'yyyyMMdd' to date format with spark session (spark) - not spark context.

Since I'm not working with spark context (sc), I cannot use the following code, although it would precisly do what I'd like it to do:

.withColumn("column1",DF.to_date(F.col("column1"),"yyyyMMdd"))

As I do not want to convert the column to timestamp, I also don't want to use the following code:

.withColumn("column1", unix_timestamp(col("column1"), "yyyyMMdd").cast("timestamp"))

The final goal is to replace the former string column by the date formatted column.

Many thanks in advance!

Why can't you use the first option, did you try running it ? — Vaibhav, Mar 26 '19 at 10:25

score 1 · Answer 1 · answered Mar 26 '19 at 12:30

1

the following code works fine:

.withColumn("column1", to_date(DF["column1"], 'yyyyMMdd'))

Thanks for your attention!

answered Mar 26 '19 at 12:30

C.Tomas

1 Answers1