I have written a code where I am listing the files in a S3 location and then in the loop copying those files to a temp folder for processing. After processing, I then move these files to an archive location.
This code-logic gets executed frequently in a cronjob.
Lately, my code is failing because in subsequent executions of the code, it somehow finds some files (which were moved in the previous execution) while listing them, but fails while trying to copy since the files don't actually exists.
The error I get is - A client error (404) occurred when calling the HeadObject operation: Key {some-file} does not exists.
Can someone please help me understand why I am facing this issue and how to resolve it?
Any help will be greatly appreciated.