0

I've posted about this question before, and I thought I had it solved, but unfortunately I still can't figure out what's going on.

In a nutshell: I have one JSON file in an s3 bucket which is updated daily. My crontab script (set to run once daily) downloads the file from my s3 bucket to my local directory, overwriting the existing file on my local directory. The script is run locally, on Mac terminal. The intention is to run the script once a day so that the file is constantly overwritten and updated.

I have verified that the JSON file in the s3 bucket is being continually updated, so I don't think that's the problem.

Here's the command, on crontab:

03 23 * * * /Library/Frameworks/Python.framework/Versions/3.7/bin/aws s3 cp s3://sfbucket.bucket/sf_events.json /Users/Documents/TownSounds_Javascript/data/sf_events.json >> /Users/Documents/logs3.txt 2>&1

The above crontab script is failing to run. The only way I can get it to run is by editing the crontab script once, manually: * * * * *

Whereupon it runs one time, and then reverts to not running at all.

When I check the logs3.txt, the date modified does not appear to be updating, which is also an indication of it's failure to run.

I've further investigated, and created a test crontab script:

* * * * * echo `date` >> /Users/Documents/testCrontab.txt

And it works fine, outputting exactly as expected.

Can anyone tell me why the crontab script for the s3 file may be failing to run? Do I need to be logged in to s3 for the script to be able to do it's thing and pull the file? Amazon s3 automatically logs me out after a period of inactivity, so this seems infeasible.

I've read that aws s3 sync might be what I'm looking for?

halfer
  • 18,701
  • 13
  • 79
  • 158
DiamondJoe12
  • 1,247
  • 2
  • 17
  • 41

1 Answers1

1

There are many reasons why an AWS CLI command may fail (in this case, aws s3 cp)... you (or the script that is running that command, e.g. crontab ) need to have appropriate credentials to interact with your aws resources (see: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). You need to check that the ACCESS_KEY & SECRET_KEY are available either as an ENV variable or you can provide it directly to the aws command (see: passing access and secret key aws cli)

mninoruiz
  • 26
  • 4
  • mninoruiz - thank you, I've followed your directions and put in my access key, secret access key, and region name. Provided I do that, should the crontab script I have above work? I should be using "aws s3 cp" command, correct? – DiamondJoe12 Feb 08 '20 at 04:54
  • 1
    Try! aws S3 cp and sync perform different functions, check online for the explanation... so it depends what you want to do, but the credentials world be the same – mninoruiz Feb 08 '20 at 05:01
  • thank you so much. You are the first person who has been able to solve this. It's working now, all I needed was to run my credentials. – DiamondJoe12 Feb 08 '20 at 20:38