Stack Overflow
Stack Overflow

Questions tagged [boto3]

Boto 3 - The Amazon Web Services (AWS) SDK for Python

About

Boto is the Amazon Web Services (AWS) software development kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

Links

5549 questions
283
votes
2 answers

Difference in boto3 between resource, client, and session?

I am using Python 2.7.12 in Ubuntu 16.04 LTS. I'm learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their…
shiva
  • 3,483
  • 5
  • 16
  • 37
260
votes
17 answers

Listing contents of a bucket with boto3

How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")? Doing the following: import boto3 s3 = boto3.resource('s3') my_bucket = s3.Bucket('some/path/') returns: s3.Bucket(name='some/path/') How do I see its contents?
Amelio Vazquez-Reina
  • 74,000
  • 116
  • 321
  • 514
243
votes
10 answers

How to handle errors with boto3?

I am trying to figure how to do proper error handling with boto3. I am trying to create an IAM user: def create_user(username, iam_conn): try: user = iam_conn.create_user(UserName=username) return user except Exception as e: …
SQDK
  • 3,167
  • 3
  • 16
  • 16
222
votes
23 answers

check if a key exists in a bucket in s3 using boto3

I would like to know if a key exists in boto3. I can loop the bucket contents and check the key if it matches. But that seems longer and an overkill. Boto3 official docs explicitly state how to do this. May be I am missing the obvious. Can anybody…
Prabhakar Shanmugam
  • 3,916
  • 5
  • 20
  • 30
196
votes
7 answers

boto3 client NoRegionError: You must specify a region error only sometimes

I have a boto3 client : boto3.client('kms') But it happens on new machines, They open and close dynamically. if endpoint is None: if region_name is None: # Raise a more specific error message that will give #…
WebQube
  • 6,772
  • 9
  • 40
  • 76
175
votes
5 answers

Open S3 object as a string with Boto3

I'm aware that with Boto 2 it's possible to open an S3 object as a string with: get_contents_as_string() Is there an equivalent function in boto3 ?
Gahl Levy
  • 2,551
  • 2
  • 10
  • 7
159
votes
4 answers

How to choose an AWS profile when using boto3 to connect to CloudFront

I am using the Boto 3 python library, and want to connect to AWS CloudFront. I need to specify the correct AWS Profile (AWS Credentials), but looking at the official documentation, I see no way to specify it. I am initializing the client using the…
Nader A. Jabbar
  • 1,778
  • 2
  • 10
  • 6
155
votes
11 answers

Save Dataframe to csv directly to s3 Python

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using…
user2494275
  • 1,553
  • 2
  • 10
  • 4
148
votes
1 answer

What is the difference between the AWS boto and boto3

I'm new to AWS using Python and I'm trying to learn the boto API however I noticed that there are two major versions/packages for Python. That would be boto and boto3. What is the difference between the AWS boto and boto3 libraries?
Matt
  • 2,832
  • 4
  • 18
  • 23
144
votes
15 answers

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

When I simply run the following code, I always gets this error. s3 = boto3.resource('s3') bucket_name = "python-sdk-sample-%s" % uuid.uuid4() print("Creating new bucket with name:", bucket_name) s3.create_bucket(Bucket=bucket_name) I have saved my…
d-_-b
  • 2,996
  • 5
  • 21
  • 36
144
votes
7 answers

How to save S3 object to a file using boto3

I'm trying to do a "hello world" with new boto3 client for AWS. The use-case I have is fairly simple: get object from S3 and save it to the file. In boto 2.X I would do it like this: import boto key =…
Vor
  • 27,623
  • 37
  • 119
  • 184
131
votes
7 answers

How to write a file or data to an S3 object using boto3

In boto 2, you can write to an S3 object using these methods: Key.set_contents_from_string() Key.set_contents_from_file() Key.set_contents_from_filename() Key.set_contents_from_stream() Is there a boto 3 equivalent? What is the boto3 method for…
jkdev
  • 9,037
  • 14
  • 52
  • 75
122
votes
5 answers

How to specify credentials when connecting to boto3 S3?

On boto I used to specify my credentials when connecting to S3 in such a way: import boto from boto.s3.connection import Key, S3Connection S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY ) I could then use S3 to…
Robert Brax
  • 4,735
  • 8
  • 32
  • 60
109
votes
16 answers

Retrieving subfolders names in S3 bucket from boto3

Using boto3, I can access my AWS S3 bucket: s3 = boto3.resource('s3') bucket = s3.Bucket('my-bucket-name') Now, the bucket contains folder first-level, which itself contains several sub-folders named with a timestamp, for instance 1456753904534. I…
mar tin
  • 6,786
  • 18
  • 60
  • 86
105
votes
9 answers

How to import a text file on AWS S3 into pandas without writing to disk

I have a text file saved on S3 which is a tab delimited table. I want to load it into pandas but cannot save it first because I am running on a heroku server. Here is what I have so far. import io import boto3 import os import pandas as…
alpalalpal
  • 1,303
  • 2
  • 11
  • 14
1
2 3
…
99 100