I'm trying to output an RDD that has ~5,000,000 rows as a text file using PySpark. It's taking a long time, so what are some tips on how to make the .saveAsTextFile() faster?
The rows are 3 columns each, and I'm saving to HDFS.
Asked
Active
Viewed 85 times
0
cshin9
- 1,150
- 4
- 15
- 30
-
1Could you give us an idea of how long it's taking, on what kind of cluster and job configuration? How big are the rows? – DNA May 12 '16 at 22:05
-
Are you saving it into `HDFS`? – Alberto Bonsanto May 12 '16 at 22:17
1 Answers
1
Without knowing how long a long time is or knowing the size of each individual row or the dimensions of the cluster, I can only make a couple of guesses.
First, in general, Spark will output a single file per partition. If your RDD is a single (or few) partitions then outputting to HDFS or GCS will appear to be slow. Consider repartitioning before outputting (repartition will also take time, if you can work the repartition into the pipeline so that it does useful work all the better). You can always call RDD#getNumPartitions to see how many partitions are in an RDD and reparatition intelligently if needed.
The second possibility I can think of is that that your HDFS could be under provisioned (e.g., out of space) or having issues causing errors that are not being surfaced very well. I would expect any HDFS write errors to be visible to the driver, but may be in container logs.
Angus Davis
- 2,584
- 10
- 20