AWS Glue Multi-column rename on DynamicFrames

Question

Documentation doesn't specify if this is allowed or not however I can't seem to get it to work and it isn't very clean to chain multiple DF's over and over.

E.g.

    df1= RenameField.apply(frame = df, old_name = "col1",new_name = "COL1")
df2= RenameField.apply(frame = df1, old_name = "col2", new_name = "COL2")

I tried a few variants and based on other Glue transforms thought the following would have worked..

df1 = RenameField.apply[(frame = df, old_name = "col1",new_name = "COL1"),
                       (frame = df, old_name = "col2", new_name = "COL2")]

score 0 · Answer 1 · answered Aug 01 '18 at 01:54

You can write clean chain code if you accept conversion like ... "DynamicFrame -> DataFrame -> DynamicFrame". DynamicFrame class has conversion methods; toDF and fromDF. I mean, it's better to do renaming on Dataframe class. In order to use PySpark Dataframe class, you can use conversion methods of toDF & fromDF.

PySpark Dataframe class has several column renaming methods, see How to change dataframe column names in pyspark?

score 0 · Answer 2 · answered Aug 03 '18 at 01:22

You can use applyMapping method from DynamicFrame to rename columns and/or cast to another data type (Scala):

val mappedDynamicFrame = sourceDynamicFrame.applyMapping(
      mappings = Seq(
        ("col1", "string", "column_1", "string"),
        ("col2", "string", "column_2", "string"),
        ("col3", "long", "column_3", "timestamp")
      ),
      caseSensitive = false,
      transformationContext = s"mapped-source"
    )

AWS Glue Multi-column rename on DynamicFrames

2 Answers2