86

What's the difference between spark.sql.shuffle.partitions and spark.default.parallelism?

I have tried to set both of them in SparkSQL, but the task number of the second stage is always 200.

user4157124
  • 2,452
  • 12
  • 22
  • 36
Edison
  • 865
  • 1
  • 7
  • 7

2 Answers2

105

From the answer here, spark.sql.shuffle.partitions configures the number of partitions that are used when shuffling data for joins or aggregations.

spark.default.parallelism is the default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set explicitly by the user. Note that spark.default.parallelism seems to only be working for raw RDD and is ignored when working with dataframes.

If the task you are performing is not a join or aggregation and you are working with dataframes then setting these will not have any effect. You could, however, set the number of partitions yourself by calling df.repartition(numOfPartitions) (don't forget to assign it to a new val) in your code.

To change the settings in your code you can simply do:

sqlContext.setConf("spark.sql.shuffle.partitions", "300")
sqlContext.setConf("spark.default.parallelism", "300")

Alternatively, you can make the change when submitting the job to a cluster with spark-submit:

./bin/spark-submit --conf spark.sql.shuffle.partitions=300 --conf spark.default.parallelism=300
Shaido
  • 22,716
  • 18
  • 57
  • 64
  • 6
    Any advice on what this number should be...? – CpILL Mar 21 '19 at 23:08
  • 1
    @CpILL: It depends on your situation, some more information and useful links can be found here: https://stackoverflow.com/questions/35800795/number-of-partitions-in-rdd-and-performance-in-spark. In particular, the Spark [documentation on tuning](https://spark.apache.org/docs/latest/tuning.html#level-of-parallelism) recommends 2-3 tasks per CPU core in the cluster. – Shaido Mar 22 '19 at 01:21
  • The default number of partition you can decide by available ram across the node that you can provide to executors. So here is the simple formula no. of partition =(available RAM in MB)/256 that means 256MB per partition. – Amit khandelwal Aug 07 '19 at 00:59
  • 3
    A little confused here. spark.sql.shuffle.partitions configures the partitions used for joins or aggregations. You then say spark.default.parallelism is used for transformations like join, reduceByKey. Arent those joins or aggregations as well? – vi_ral Jun 01 '20 at 14:50
  • @Shaido Even if I apply this --conf spark.sql.shuffle.partitions=300 , still I see a lot of stages being generated and most of them does not have records , which is leading to Jave heap OOM exception , how to handle this situation? – BdEngineer Dec 18 '20 at 04:43
  • @CpILL you can see some formulas for setting these parameters here: https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/ – Scott Mar 10 '21 at 19:54
13

spark.default.parallelism is the default number of partition set by spark which is by default 200. and if you want to increase the number of partition than you can apply the property spark.sql.shuffle.partitions to set number of partition in the spark configuration or while running spark SQL.

Normally this spark.sql.shuffle.partitions it is being used when we have a memory congestion and we see below error: spark error:java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE

so set your can allocate a partition as 256 MB per partition and that you can use to set for your processes.

also If number of partitions is near to 2000 then increase it to more than 2000. As spark applies different logic for partition < 2000 and > 2000 which will increase your code performance by decreasing the memory footprint as data default is highly compressed if >2000.

Amit khandelwal
  • 385
  • 3
  • 5