2

Is there a way other than repartition(which slows the processing) to combine all 1mb files into multiple big files?

run spark code on 500Gb of data , on 100 executers 24 cores each, but save them into large files with 128mb each. now it is saving 1 mb each file.

spark.sql("set pyspark.hadoop.hive.exec.dynamic.partition=true")

spark.sql("set pyspark.hadoop.hive.exec.dynamic.partition.mode=nonstrict")

spark.sql("set hive.exec.dynamic.partition=true")

spark.sql("set hive.exec.dynamic.partition.mode=nonstrict")

spark.sql("set hive.merge.tezfiles=true")

spark.sql("SET hive.merge.sparkfiles = true")

spark.sql("set hive.merge.smallfiles.avgsize=128000000")

spark.sql("set hive.merge.size.per.task=128000000")
MERN
  • 2,842
  • 6
  • 13
  • 34
manohar g
  • 33
  • 3

1 Answers1

1

Option-1:

You can do .coalesce(n)(no shuffle will happen) on your dataframe and then use .option("maxRecordsPerFile",n) to control the number of records written in each file.

Option-2:

Using spark.sql.shuffle.partitions=n this option is used to control the number of shuffles happens.

Then use df.sort("<col_name>").write.etc will create exactly the number of files that we mentioned for shuffle.partitions.

Option-3:

Hive:

Once the spark job is done then trigger hive job insert overwrite by selecting the same table and use sortby,distributedby,clusteredby and set the all hive configurations that you have mentioned in the question.

Insert overwrite table select * from table sort by <col1> distributed by <col2>

Option-4:

Hive:

If you have ORC table then schedule concatenate job to run periodically 

alter table <table_name> concatenate;

If none of the methods seems to be feasible solutions then .repartition(n) will be the way to go as this will take extra overhead but we are going to end up ~evenly sized files in HDFS and boost up the performance while reading these files from hive/spark.

Shu
  • 22,644
  • 2
  • 17
  • 38