for example, i have a folder:
/
- test.py
- test.yml
and the job is submited to spark cluster with:
gcloud beta dataproc jobs submit pyspark --files=test.yml "test.py"
in the test.py
, I want to access the static file I uploaded.
with open('test.yml') as test_file:
logging.info(test_file.read())
but got the following exception:
IOError: [Errno 2] No such file or directory: 'test.yml'
How to access the file I uploaded?