Stack Overflow
Stack Overflow

Questions tagged [s3fs]

s3fs is a FUSE filesystem which allows to mount an Amazon S3 bucket as a local filesystem.

s3fs allows Linux and macOS to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, allowing use of other tools like s3cmd.

s3fs is stable and is being used in number of production environments, e.g., rsync backup to s3.

Important Note: Your kernel must support FUSE, kernels earlier than 2.6.18-164 may not have FUSE support (see issue #140 ). Virtual Private Servers (VPS) may not have FUSE support compiled into their kernels.

There is also a Python library called s3fs. For questions related to this library, please use .

194 questions
6
votes
2 answers

s3fs with aws ec2 instance and using instance profiles

As far as I can tell the only way to mount an s3 bucket with s3fs is to use an accesskey:secretkey specified in a file with various file locations supported. However, if I'm an ec2 instance, in the local s3 account, with an instance profile, I…
user2615236
  • 481
  • 1
  • 4
  • 8
6
votes
5 answers

Mount multiple s3fs buckets automatically with /etc/fstab

In the s3fs instruction wiki, we were told that we could auto mount s3fs buckets by entering the following line to /etc/fstab s3fs#mybucket /mnt/mybucket fuse allow_other,use_cache=/tmp,url=https://s3.amazonaws.com 0 0 This works fine for 1…
Ryan Cheng
  • 91
  • 1
  • 1
  • 5
5
votes
1 answer

Mount s3fs as docker volume

So I just want to add my s3 bucket from amazon to my docker swarm. I've saw many "possible" solutions on the internet but I can't connect them to add the content of my bucket as volume. So the last thing I've tried was the command statet here (Is…
thmspl
  • 2,052
  • 2
  • 13
  • 36
5
votes
4 answers

How could I store uploaded images to AWS S3 on PHP

I'm on an EC2 instance and I wish to connect my PHP website with my Amazon S3 bucket, I already saw the API for PHP here: http://aws.amazon.com/sdkforphp/ but it's not clear. This is the code line I need to edit in my controller: …
NineCattoRules
  • 1,626
  • 5
  • 30
  • 65
5
votes
1 answer

s3fs on Amazon EMR: Will it scale for approx 100million small files?

Please refer to the following questions already asked: Write 100 million files to s3 and Too many open files in EMR The size of data being handled here is atleast around 4-5TB. To be precise - 300GB with gzip compression. The size of input will grow…
Kartikeya Sinha
  • 508
  • 1
  • 5
  • 19
5
votes
2 answers

Heroku: Using external mount in local filesystem

I know it's possible to mount an Amazon S3 bucket using Fuse (s3fs [or s3fsr ruby gem?]). My case is specific to Heroku. Heroku's filesystem is readonly for scalability and such, but is there a way to mount an amazon s3 in Heroku's filesystem? In my…
Jim
  • 954
  • 9
  • 21
4
votes
2 answers

S3 buckets randomly unmount on EC2 instance

My company currently is using S3fs and Ec2 from AWS. We have been mounted our s3 buckets on our Ec2 instances, but after some time (a week, for example) some of the buckets unmount by themselves and our server instances become nearly useless. The…
fiction
  • 10,777
  • 5
  • 45
  • 72
4
votes
1 answer

Versioning file system with Amazon S3 as backend

I'm trying to make the following work on my Debian computers and one OS X Laptop. What I would like to have is some kind of versioning file system that uses Amazon S3 as a backend. What I was thinking is to use s3fs (using FUSE) to mount the bucket,…
Jonas Teuwen
  • 211
  • 2
  • 7
3
votes
3 answers

s3fs: AWS Message: Access Denied Ubuntu 11.10

i installe s3fs as it is described here http://code.google.com/p/s3fs/wiki/InstallationNotes then in i create user bucket_user then put his accessKeyId:secretAccessKey in /etc/passwd-s3fs them is S3 i create a bucket super_bucket and set its…
fullpipe
  • 348
  • 4
  • 14
3
votes
1 answer

Using Amazon S3 php REST API vs. Mounting S3 bucket to server (s3fs)

I will be launching an application in the very near future which will, in part, require users to upload files (images) to be viewed by other members. I like the idea of S3 as it is relatively cheap and scales automatically. My problem is how I will…
minjoon
  • 41
  • 5
3
votes
1 answer

s3fs volume exposed via nginx

I am mounting S3 bucket to Ubuntu VM with command root@factory:~# s3fs my-bucket /mnt/s3-bucket -o passwd_file=/etc/.passwd-s3fs -o noatime -o allow_other -o uid=1000 -o gid=1000 -o use_cache=/tmp -o default_acl=public-read-write This volume gets…
roy
  • 4,437
  • 16
  • 58
  • 118
3
votes
2 answers

Amazon AWS s3fs mount problem on Fedora 14

I successfully compiled and installed s3fs (http://code.google.com/p/s3fs/) on my Fedora 14 machine. I included the password credentials in /etc/ as specified in the guide. When I run: sudo /usr/bin/s3fs bucket_name /mnt/bucket_name/ it runs…
Alex
  • 616
  • 1
  • 12
  • 26
3
votes
1 answer

What's the fastest way to use Ceph volumes in Docker Swarm?

I want to set up a Swarm with persistent and replicated volumes through Ceph. I see these options to combine both services, once both are set up: Configure the host OS to mount a CephFS in /var/lib/docker/volumes. Use rexray/rbd as a volume…
Yajo
  • 4,116
  • 2
  • 24
  • 33
3
votes
2 answers

best way to migrate billions of files on a single partition in a data center to s3?

We have a data center with a 10G direct connect circuit to AWS. In the data center, we have an IBM XIV storage infrastructure with GPFS filesystems containing 1.5 BILLION images (about 50k each) in the single top level directory. We could argue all…
godeatgod
  • 77
  • 1
  • 5
3
votes
3 answers

Set Cache Limit or Expiration for S3FS

I have this option "-o use_cache=/tmp" set when I mount my S3 bucket. Is there a limit on how much room it will try to use in tmp? Is there a way to limit that or otherwise expire items after X amount of time?
Jafo
  • 913
  • 1
  • 9
  • 18
1
2
3
…
12 13