How many number of partitions will be there if I have 100 gb of data in spar? If someone says we have 20 will that be enough for the memory
Asked
Active
Viewed 26 times
1
-
Hi Prateek, wellcome to SO. Please, read this link. https://stackoverflow.com/questions/42171499/get-current-number-of-partitions-of-a-dataframe – aironman Sep 05 '20 at 19:10
1 Answers
0
The number of partitions depends on the data source in use. For file-based data sources that use HDFS as a storage there will be as many partitions as there are blocks. Other data sources can report this requirement differently (based on their own internal logic).
Jacek Laskowski
- 64,943
- 20
- 207
- 364