Read file data from aws s3 pyspark

Question

I have a json file placed in s3. The s3 url is similar to the below one:

https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json

But in pyspark when pass the same, It is not reading the file.

path = "https://s3-eu-region-1.amazonaws.com/dir-resources/sample.json"
df=spark.read.json(path)

But I am able to download it through browser.

Did you added configuration for S3? https://stackoverflow.com/a/30852341/3945375 — Gocht, Mar 13 '18 at 22:48

score 0 · Answer 1 · answered Mar 11 '18 at 16:32

0

Assuming that dir-resources is the name of your bucket, you should be able to access to the file with the following URI:

path = "s3://dir-resources/sample.json"

In some cases, you may have to use the s3n protocol instead:

path = "s3n://dir-resources/sample.json"

answered Mar 11 '18 at 16:32

carrdelling

Is region not relevant here? – Tom J Muthirenthi Mar 11 '18 at 16:35
In my case it isn't - I can access S3 files from pyspark with just the bucket name. Is this working for you? – carrdelling Mar 11 '18 at 16:36
1

no. I am getting the error `Py4JJavaError: An error occurred while calling o32.json. : java.io.IOException: No FileSystem for scheme: s3n at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:26` – Tom J Muthirenthi Mar 11 '18 at 16:38
Then it may be a problem with your local Spark installation. Have a look at this other question: https://stackoverflow.com/questions/30851244/spark-read-file-from-s3-using-sc-textfile-s3n – carrdelling Mar 11 '18 at 16:42
Tried and installed. It didn't help – Tom J Muthirenthi Mar 11 '18 at 19:46

1 Answers1