4

Using the Java S3 SDK, I can create buckets and put objects in them and list contents and delete buckets. The only thing I cannot do is create the same bucket again after deleting it.

In the stack trace shown below, I have previously created a bucket named gormanm-0709-r-o-o-t, used it, then deleted it via s3.deleteBucket("gormanm-0709-r-o-o-t"). I then attempt to recreated it via s3.createBucket("gormanm-0709-r-o-o-t", "us-standard") but get the error shown below.

com.amazonaws.services.s3.model.AmazonS3Exception: Container gormanm-0709-r-o-o-t exists (Service: Amazon S3; Status Code: 409; Error Code: BucketAlreadyExists; Request ID: 1be8b569-3db4-4eff-bf1e-c4b8dac20272), S3 Extended Request ID: null
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4187)
 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4134)
 at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:1021)
 at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:973)
 at com.ibm.saas.file.cos.CosFile.createBucket(CosFile.java:794)
 at com.ibm.saas.file.cos.CosFile.ensureBucketExists(CosFile.java:781)
 at com.ibm.saas.file.cos.CosFile.mkdirs(CosFile.java:665)
 at com.ibm.saas.file.cos.CosFileServiceTest.testDir(CosFileServiceTest.java:148)
 at com.ibm.saas.file.cos.CosFileServiceTest.testRootOperations(CosFileServiceTest.java:82)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
 at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

I know for sure I deleted it (as the code did not throw any exception and I don't see the bucket appearing in the Bluemix/Softlayer console) and I know for sure no one else "jumped on" my bucket name since I have done this testing over and over again.

My guess is that there is some period of time before the bucket name becomes available for use again but I'm only guessing about that and, even if that were true, there should be some documentation somewhere explaining the rules.

Speaking of documentation, the Getting Help page says to go here (https://developer.ibm.com/answers/smartspace/public-cloud-object-storage/) for questions but that page does not exist.

ralphearle
  • 1,698
  • 11
  • 18
Mike Gorman
  • 169
  • 10
  • And, strangely, sometimes the error message is a little different. Exact same unit test running as shown above but this time the error isn't "Container xxx exists", it's "The requested bucket name is not available": – Mike Gorman Mar 26 '17 at 11:36
  • com.amazonaws.services.s3.model.AmazonS3Exception: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. (Service: Amazon S3; Status Code: 409; Error Code: BucketAlreadyExists; Request ID: 4fafce9f-8852-4ec4-a1b7-57be9fb7fff7), S3 Extended Request ID: null – Mike Gorman Mar 26 '17 at 11:37
  • UPDATE: Waited one hour and ran the exact same test again and, this time, it worked (but now, after deleting that bucket, I now have to wait xxx more minutes before it's name is available for use again). So there definitely does seem to be some timeout period before bucket names can be reused but, again, no documentation that says anything about it. Can anyone confirm or deny? – Mike Gorman Mar 26 '17 at 12:37
  • 2
    This is a bucket in IBM Cloud Object Storage, not S3, right? Either way, bucket deletion is likely a low priority operation on the back-end, requires coordination across multiple, distributed locations, and has DNS consequences so it should not be a surprise that it's not instantaneous. – jarmod Mar 26 '17 at 15:17
  • 1
    About the Getting Help link, the page for questions about Cloud Object Storage and its capabilities is https://developer.ibm.com/answers/smartspace/cloudobjectstorage/. That link is currently correct in the documentation. – ralphearle Mar 29 '17 at 15:43

2 Answers2

4

In IBM COS there is a 10-minute period after bucket deletion where the name is reserved. This delay prevents possible caching issues where a command might write to a deleted bucket. The chances of this are small, and we might shrink this window down a bit, but generally speaking it's not a great practice to be deleting and recreating buckets. Is there a particular reason you need to be able to do this in your workflow?

And yes, as mentioned by @jarmod in the top level comments, this behavior is due to the large number distributed logical appliances that control the slicing and reassembly of objects. The bucket deletion is still mostly immediately consistent, as writes and listings will be not be possible upon receiving the 204 response, but this behavior could be better documented.

While Michael's answer refers to AWS S3, the information is more or less accurate for our implementation as well.

Nick Lange
  • 777
  • 4
  • 7
  • 1
    Thanks. So is my rationale correct - that IBM COS is a completely different, ground-up implementation of "cloud storage" that JUST SO HAPPENS to use the same S3 API as AWS and, thus, IBM COS implementation details and side-effects might be completely different than AWS? Or is IBM COS back-end implementation BUILT ON TOP OF AWS S3 (in which case I should expect any behavior discussed in AWS documentation to apply to IBM COS)? – Mike Gorman Mar 27 '17 at 12:15
  • BTW - to answer your question about why I might be deleting buckets and re-creating them with the same name, it is simply because I am creating an vendor-neutral client-side API that can work with any cloud storage implementation so I naturally have APIs that front-end createBucket and deleteBucket so (a) I have no control over what my developers might do and (b) just running simple unit tests where I have to "setup" and "teardown" in between each and every test, part of that setup & teardown includes creating or deleting some bucket. – Mike Gorman Mar 27 '17 at 12:18
  • Since I was using the same bucket name for each of my unit tests, my first test would always pass and all subsequent tests would always fail. – Mike Gorman Mar 27 '17 at 12:18
  • Great question. IBM COS is built on software developed by Cleversafe, which was acquired by IBM in late 2015. The storage engine itself is completely different from S3 but in the public cloud the top edge interface layer uses an S3-compatible API. This is mainly because S3 is the closest thing there is to a "standard" object storage API and it's familiar to many developers. So yes, details and side effects may be different, but anything that is application-facing should look and feel the same with the intention that compatibility with S3 tools and SDKs is preserved. – Nick Lange Mar 28 '17 at 13:25
  • 1
    And I totally understand your situation - just yesterday we ran into the other end of that problem. In order to get around the cool down period we tried creating new buckets for each test and then ran into the limit of 100 buckets per account. :( – Nick Lange Mar 28 '17 at 13:25
3

From the S3 Developer Guide:

Bucket ownership is not transferable; however, if a bucket is empty, you can delete it. After a bucket is deleted, the name becomes available to reuse, but the name might not be available for you to reuse for various reasons. For example, some other account could create a bucket with that name. Note, too, that it might take some time before the name can be reused. So if you want to use the same bucket name, don't delete the bucket. [emphasis added]

http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

There is your documentation of the expected delay, and a bonus nugget: you really should not be deleting and recreating buckets. Since the bucket namespace is global, only one bucket of a given name can exist at any given time, and it can of course only be in one region. Buckets are only intended to be deleted when you're never going to need them again -- otherwise, empty the bucket of objects, but keep the bucket. Someone might snap the name up, if it's particularly non-obscure. Also:

The high-availability engineering of Amazon S3 is focused on get, put, list, and delete operations. Because bucket operations work against a centralized, global resource space, it is not appropriate to create or delete buckets on the high-availability code path of your application. It is better to create or delete buckets in a separate initialization or setup routine that you run less often.

http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Although the details are not public, all S3 regions are in some sense aware of all buckets. If you send an S3 request to the wrong region, that wrong region does know the correct region and will return an error that includes an (apparently undocumented) x-amz-bucket-region: response header.

So when buckets are created -- regardless of region -- there is necessarily some internal communication with the S3 global database that occurs when creating a bucket. A disruption here could presumably prevent the creation of a bucket in any region, even if that region was otherwise operating normally.

Note that -- as of this writing -- another section of the documentation, the S3 Getting Started Guide, provides essentially the same information; however, until at least late 2015, the information provided there was different.

When you delete a bucket, there may be a delay of up to one hour before the bucket name is available for reuse in a new region or by a new bucket owner. If you re-create the bucket in the same region or with the same bucket owner, there is no delay.

https://web.archive.org/web/20150905132716/http://docs.aws.amazon.com/AmazonS3/latest/gsg/DeletingAnObjectandBucket.html

Intuitively, I suspect that the continued growth of S3 could have made this information obsolete prior to its removal, and prior this snapshot from 2015.

Community
  • 1
  • 1
Michael - sqlbot
  • 139,456
  • 21
  • 252
  • 328
  • Thanks. I didn't look at AWS documentation because I'm using IBM Cloud Object Storage that, though adopts the same S3 API, is its own implementation and what I was experiencing wasn't a question of API but a question of implementation but your explanation of how AWS works is what I guessed was going on with IBM COS as well, that bucket deletion and name re-use was subject to some thawing period, for all the reasons mentioned above. – Mike Gorman Mar 27 '17 at 12:13