44

aws s3 sync --delete removed some new files.

For example:

There is a file in the bucket - S3://my-bucket/images/1.jpg

Then, I uploaded a file to the server: 2.jpg

There are 2 files in the server: 1.jpg and 2.jpg

Start running the sync cronjob:

aws s3 sync s3://my-bucket/ ./ --delete
aws s3 sync  ./ s3://my-bucket/ --delete

Why do we add --delete - we want to delete the files in s3 and sync it to the server.

We will upload files to the server and remove the files in s3.

Is there any way to fix it?

John Rotenstein
  • 165,783
  • 13
  • 223
  • 298
Keith Kong
  • 541
  • 1
  • 4
  • 3

3 Answers3

81

By default, the aws sync command (see documentation) does not delete files. It simply copies new or modified files to the destination.

Using the --delete option deletes files that exist in the destination but not in the source.

So, if your source contains: 1.jpg and 2.jpg and the destination contains 1.jpg, 2.jpg and 3.jpg, then using the --delete option will delete 3.jpg from the destination.

I see that you are running the sync command in both directions. Your first command (that syncs S3 to a local directory) will delete any local files that are not in S3.

If your goal is to copy all local files to S3 and all S3 files to the local directory, without deleting any files, then do not use the --delete option.

George
  • 2,612
  • 16
  • 27
John Rotenstein
  • 165,783
  • 13
  • 223
  • 298
  • Hi, My goal is to delete some old files in s3 bucket, then I run that command with --delete and I want it to remove my local files as well. And Then I can upload some new files from local to s3 with aws s3 sync command. Is it possible to do that? – Keith Kong Jun 08 '15 at 02:22
  • 3
    Yes, you can do that. Just be aware that `--delete` will remove files in the destination that are not in the source. So, only use it from the "master" location. It will not remove files from the "local" (source) location. Just play around with the order in which you run the commands to get the desired behaviour. – John Rotenstein Jun 09 '15 at 04:09
  • I was kinda expecting the default behaviour of a sync command to do this, as that would make it different from the cp command. But testing. – James Woolfenden Jun 26 '19 at 08:18
4

--delete option to remove files or objects from the target not present in the source.

sarat chandra
  • 81
  • 1
  • 3
1

For the new s3cmd you can use the option --delete-removed to remove extra files in the destination, that are not present the source.

Seff
  • 1,048
  • 14
  • 14