134

How can I create a folder under a bucket using boto library for Amazon s3?

I followed the manual, and created the keys with permission, metadata etc, but no where in the boto's documentation it describes how to create folders under a bucket, or create a folder under folders in bucket.

Eric Leschinski
  • 123,728
  • 82
  • 382
  • 321
vito huang
  • 3,482
  • 6
  • 21
  • 21
  • it seems amazon s3 doesn't have concept of folder, some suggest creating key with name like "folder/test.txt' to get around it. i tried using firefox s3 plugin to create folder, and list all keys in boto, it outputs the folder i just created as "], so how can i view/add/modify content to/from this folder? – vito huang Dec 21 '09 at 12:29
  • Note: the AWS S3 management interface option for creating "folders" is not compatible with S3FS i.e. create a "folder" using the interface and try to get a listing of said folder through S3FS mount point. – jldupont Feb 17 '13 at 12:49
  • Note2: creating a "folder" through S3FS is compatible with AWS S3 Management interface though. – jldupont Feb 17 '13 at 12:52
  • See @JaHax answer below for an example of how to do this with Boto. Easy! – e.thompsy Jan 16 '15 at 02:28

11 Answers11

141

There is no concept of folders or directories in S3. You can create file names like "abc/xys/uvw/123.jpg", which many S3 access tools like S3Fox show like a directory structure, but it's actually just a single file in a bucket.

Rahul K P
  • 9,913
  • 3
  • 28
  • 46
user240469
  • 1,514
  • 1
  • 10
  • 5
  • 1
    thanks for the answer, so i guess if i want to see the contents of a particular folder, i will need to traverse lots other unnecessary files? – vito huang Jan 27 '10 at 23:50
  • The S3 API lets you query a bucket for keys with a given prefix. So you should be able to construct a request that only returns objects in your logical folder. See: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?ListingKeysHierarchy.html – James Cooper Jul 22 '10 at 19:58
  • 7
    there is very well defined concept of folders. the answer is incorrect.. see elranu answer below. – Boppity Bop Mar 28 '14 at 09:49
  • 21
    @BoppityBop: There is **no concept of folders** in **S3**. S3 does not have folders, even though the *management console* and many tools do represent keys with slashes as such. See [Working with Folders](http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html) and read the part: *"So the console uses object key names to present folders and hierarchy.* ***In Amazon S3, you have only buckets and objects.**"* – Antti Haapala Aug 04 '14 at 18:29
  • 9
    It might not technically be a folder, but there definitely seems to be folder support. In the aws console when looking at a bucket you can click "Create Folder" and it will make one, and they can be empty, and pull meta data from them. – phazei Jul 11 '15 at 06:10
  • 5
    S3 is a giant, custom DynamoDB key-value store. Some tools (including the AWS web console) provide some functionality that mimics a directory tree, but you'll be working against S3 rather than working with it if your applications assume it's equivalent to a file system. Renaming what appears to be a directory, for instance, requires scanning through the keyspace and modifying every key containing the 'directory name' in question. On the other hand, being a key-value store, there's no need for creating 'parent directories' or cleaning 'empty folders', which mimicking a file system would require – scooter-dangle Feb 04 '16 at 18:18
  • This answer is out of date and/or inaccurate. Folders are very much a concept on S3, such as the easily observed "Create folder" button in the console. While the intention of the answer is good (in that folders aren't generally useful constructs alone), no help is provided if you have a legitimate need to create a folder with no initial content, such as cases where users may be uploading content to specially defined folders through the console. – bsplosion Feb 17 '20 at 21:13
  • 1
    if it looks like a folder, behaves like one. then why people keep saying there isnt a concept of folders in S3. How S3 manages internally is upto them. But if i could organize them with folders then it call it a folder – Ravisha Jun 29 '20 at 05:22
54

Assume you wanna create folder abc/123/ in your bucket, it's a piece of cake with Boto

k = bucket.new_key('abc/123/')
k.set_contents_from_string('')

Or use the console

TomNg
  • 1,699
  • 2
  • 16
  • 25
  • 8
    This is the correct answer. It is possible with Boto and this is how you do it. Actually the `new_key()` is all you need. The string sent to the function could be `abc/123/` or `abc/123/newfile.txt`. Whatever you prefer. I like to wrap this in an if statement after I try to get what I hope is there like this: `key = bucket.get_key(upgrade_path) if key is None: key = bucket.new_key(upgrade_path)` Essentially, if it is not there, create it! – e.thompsy Jan 16 '15 at 02:25
  • 2
    With the latest api, bucket.key('abc/123/') will also achieve the same result. – rahulmishra May 15 '15 at 10:03
  • Specifying a directory key also works with uploading files in **multipart uploads** ```mp = self._bucket.initiate_multipart_upload(bucket_key)``` – storm_m2138 Aug 09 '16 at 17:57
21

With AWS SDK .Net works perfectly, just add "/" at the end of the folder name string:

var folderKey =  folderName + "/"; //end the folder name with "/"
AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(AWSAccessKey, AWSSecretKey);
var request = new PutObjectRequest();
request.WithBucketName(AWSBucket);
request.WithKey(folderKey);
request.WithContentBody(string.Empty);
S3Response response = client.PutObject(request);

Then refresh your AWS console, and you will see the folder

elranu
  • 2,212
  • 6
  • 30
  • 54
  • Just tried this with the AWS::S3 ruby library. It creates a "folder" with a file with a blank filename in it... So it doesn't really work. Mohammad Asgari's solution works fine though! – Nico Feb 26 '12 at 11:39
  • @Nico but did you end the filename with a "/", just as the code comment says? – elranu Feb 26 '12 at 15:01
  • Yes, I added the slash. I did AWS::S3::S3Object.store('test/', '', 'my_bucket') – Nico Mar 19 '12 at 17:20
  • Issued "putObject" using empty string and slash on the end worked for me. I'm using PHP with the tpyo/amazon-s3-php-class library. – MikeMurko Jan 13 '13 at 15:37
  • of course AWS has folders. look at your S3 console. there is **button Create Folder**. this answer is the correct answer.. just make an effort.. – Boppity Bop Mar 28 '14 at 09:46
  • 2
    @BoppityBop yes it has a create folder button, but that probably just makes some sort of empty file as well – jamylak Dec 22 '15 at 00:47
18

Use this:

import boto3
s3 = boto3.client('s3')
bucket_name = "YOUR-BUCKET-NAME"
directory_name = "DIRECTORY/THAT/YOU/WANT/TO/CREATE" #it's name of your folders
s3.put_object(Bucket=bucket_name, Key=(directory_name+'/'))
Reza Amya
  • 982
  • 14
  • 33
  • Thanks for the help, I am getting below error. can you please help me .botocore.exceptions.NoCredentialsError: Unable to locate credentials – Anvesh Dec 18 '17 at 07:19
  • 1
    You need to specify your aws credentials in some way or the other. Check this out : https://boto3.readthedocs.io/en/latest/guide/configuration.html#guide-configuration – Deepak G M Jul 03 '18 at 09:05
7

Append "_$folder$" to your folder name and call put.

    String extension = "_$folder$";
    s3.putObject("MyBucket", "MyFolder"+ extension, new ByteArrayInputStream(new byte[0]), null);

see: http://www.snowgiraffe.com/tech/147/creating-folders-programmatically-with-amazon-s3s-api-putting-babies-in-buckets/

Mamad Asgari
  • 117
  • 2
  • 3
5

Tried many method above and adding forward slash / to the end of key name, to create directory didn't work for me:

client.put_object(Bucket="foo-bucket", Key="test-folder/")

You have to supply Body parameter in order to create directory:

client.put_object(Bucket='foo-bucket',Body='', Key='test-folder/')

Source: ryantuck in boto3 issue

azzamsa
  • 1,029
  • 1
  • 12
  • 18
4

It's really easy to create folders. Actually it's just creating keys.

You can see my below code i was creating a folder with utc_time as name.

Do remember ends the key with '/' like below, this indicates it's a key:

Key='folder1/' + utc_time + '/'

client = boto3.client('s3')
utc_timestamp = time.time()


def lambda_handler(event, context):

    UTC_FORMAT = '%Y%m%d'
    utc_time = datetime.datetime.utcfromtimestamp(utc_timestamp)
    utc_time = utc_time.strftime(UTC_FORMAT)
    print 'start to create folder for => ' + utc_time

    putResponse = client.put_object(Bucket='mybucketName',
                                    Key='folder1/' + utc_time + '/')

    print putResponse
Gabriel Wu
  • 1,158
  • 13
  • 22
4

Update for 2019, if you want to create a folder with path bucket_name/folder1/folder2 you can use this code:

from boto3 import client, resource

class S3Helper:

  def __init__(self):
      self.client = client("s3")
      self.s3 = resource('s3')

  def create_folder(self, path):
      path_arr = path.rstrip("/").split("/")
      if len(path_arr) == 1:
          return self.client.create_bucket(Bucket=path_arr[0])
      parent = path_arr[0]
      bucket = self.s3.Bucket(parent)
      status = bucket.put_object(Key="/".join(path_arr[1:]) + "/")
      return status

s3 = S3Helper()
s3.create_folder("bucket_name/folder1/folder2)
Ilya Vinnichenko
  • 889
  • 8
  • 3
2

Although you can create a folder by appending "/" to your folder_name. Under the hood, S3 maintains flat structure unlike your regular NFS.

var params = {
            Bucket : bucketName,
            Key : folderName + "/"
        };
s3.putObject(params, function (err, data) {});
vpage
  • 61
  • 2
0

S3 doesn't have a folder structure, But there is something called as keys.

We can create /2013/11/xyz.xls and will be shown as folder's in the console. But the storage part of S3 will take that as the file name.

Even when retrieving we observe that we can see files in particular folder (or keys) by using the ListObjects method and using the Prefix parameter.

CDub
  • 12,780
  • 4
  • 47
  • 65
Balaram
  • 9
  • 1
0

Apparently you can now create folders in S3. I'm not sure since when, but I have a bucket in "Standard" zone and can choose Create Folder from Action dropdown.

Perelx
  • 377
  • 2
  • 13