From my machine, I've configured the hadoop core-site.xml
to recognize the gs://
scheme and added gcs-connector-1.2.8.jar as a Hadoop lib. I can run hadoop fs -ls gs://mybucket/
and get the expected results. However, if I try to do the analogue from java using:
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
FileStatus[] status = fs.listStatus(new Path("gs://mybucket/"));
I get the files under root in my local HDFS instead of in gs://mybucket/
, but with those files prepended with gs://mybucket
. If I modify the conf with conf.set("fs.default.name", "gs://mybucket");
before obtaining the fs, then I can see the files on GCS.
My question is:
1. Is this expected behavior?
2. Is there a disadvantage to using this hadoop FileSystem api as opposed to the google cloud storage client api?