4

I'm new to AWS and specifically to the AWS CLI tool, but so far I seem to be going OK.

I'm using the following commands to connect to AWS S3 and synchronise a local directory to my S3 bucket:

set AWS_ACCESS_KEY_ID=AKIAIMYACCESSKEY
set AWS_SECRET_ACCESS_KEY=NLnfMySecretAccessCode
set AWS_DEFAULT_REGION=ap-southeast-2
aws s3 sync C:\somefolder\Data\Dist\ s3://my.bucket/somefolder/Dist/ --delete

This is uploading files OK and displaying the progress and result for each file. Once the initial upload is done, I'm assuming that all new syncs will just upload new and modified files and folders. Using the --delete will remove anything in the bucket that no longer exists on the local server.

I'd like to be able to output the results of each upload (or download in the case of other servers which will be getting a copy of what is being uploaded) to a .txt file on the local computer so that I can use blat.exe to email the contents to someone who will be monitoring the sync.

All of this will be put into a batch file that will be scheduled to run nightly.

Can the output to .txt be done? If so, how?

helloV
  • 42,534
  • 4
  • 100
  • 125
Reece
  • 669
  • 4
  • 19
  • 37

1 Answers1

4

I haven't tested this myself, but I found some resources that indicate you can redirect output from command-line driven applications in Windows command prompt just like you would in linux.

aws s3 sync C:\somefolder\Data\Dist\ s3://my.bucket/somefolder/Dist/ --delete > output.txt

The resources I found are:

https://stackoverflow.com/a/16713357/4471711

https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true

Once the initial upload is done, I'm assuming that all new syncs will just upload new and modified files and folders. Using the --delete will remove anything in the bucket that no longer exists on the local server.

That is correct, sync will upload either new or modified files as compared to the destination (whether it is an S3 bucket or your local machine).

--delete will remove anything in the destination (not necessarily an S3 bucket) that is not in the source. It should be used carefully so as to avoid a situation where you've downloaded, modified and sync one file and because your local machine doesn't have ALL of the files, use of the --delete flag will then delete all other files at destination.

Community
  • 1
  • 1
Brooks
  • 6,001
  • 4
  • 44
  • 75
  • not concerned about using --delete unless it modifies the source. the source is the repository for our documents and will always be the correct listing/version. this server should only ever upload (and delete) data to S3. I won't be using --delete for the batch file that does downloads FROM S3 on the other servers. Using the > works!!! Thanks. – Reece Jan 29 '16 at 04:16
  • one question regarding this method though - can it output to the console as well as file? – Reece Jan 29 '16 at 04:17
  • Not to my knowledge. I don't think that's even possible in Linux with just the output redirecting to a file. You could try working within Cygwin or, as you discovered already, maybe the AWS Powershell has some better features. – Brooks Jan 29 '16 at 04:25
  • Instead of trying to duplicate the output to the console and file in one step, can you write to the file and then output the contents of the file in a second step? You can output the contents of the file with "type output.txt". http://superuser.com/questions/256651/how-to-echo-contents-of-file-in-a-dos-windows-command-prompt – Brooks Jan 29 '16 at 04:26
  • You can send the output to both the console and the file in one step using the Linux `tee` command. https://en.wikipedia.org/wiki/Tee_(command) – Mark B Jan 29 '16 at 16:31
  • In a response to another answer (since deleted I guess), the OP mentioned this is being done in Windows cmd prompt. – Brooks Jan 29 '16 at 17:58