Stack Overflow
Stack Overflow

Questions tagged [google-cloud-python]

Use this tag if you are going to refer something from Python libraries OR API Clients of Google Cloud Platform.

183 questions
19
votes
1 answer

Efficiently write a Pandas dataframe to Google BigQuery

I'm trying to upload a pandas.DataFrame to google big query using the pandas.DataFrame.to_gbq() function documented here. The problem is that to_gbq() takes 2.3 minutes while uploading directly to Google Cloud Storage GUI takes less than a minute.…
Pablo
  • 1,876
  • 2
  • 19
  • 36
17
votes
4 answers

Google Cloud Storage + Python : Any way to list obj in certain folder in GCS?

I'm going to write a Python program to check if a file is in certain folder of my Google Cloud Storage, the basic idea is to get the list of all objects in a folder, a file name list, then check if the file abc.txt is in the file name list. Now the…
Reed_Xia
  • 920
  • 2
  • 11
  • 27
17
votes
5 answers

Google Cloud Storage Signed URLs with Google App Engine

It's frustrating to deal with the regular Signed URLs (Query String Authentication) for Google Cloud Storage. Google Cloud Storage Signed URLs Example -> Is this really the only code available in the whole internet for generating Signed URLs for…
roberto.cr
  • 302
  • 3
  • 6
14
votes
1 answer

Google Cloud Storage: How to Delete a folder (recursively) in Python

I am trying to delete a folder in GCS and its all content (including sub-directories) with its Python library. Also I understand GCS doesn't really have folders (but prefix?) but I am wondering how I can do that? I tested this code: from…
kee
  • 8,915
  • 18
  • 85
  • 149
12
votes
1 answer

What is the difference between google.cloud.pubsub_v1 and google.cloud.pubsub?

I see both used in different documentation from Google: from google.cloud import pubsub Is found…
gunit
  • 2,752
  • 2
  • 20
  • 38
11
votes
2 answers

Blob.generate_signed_url() failing to AttributeError

So I'm trying to produce temporary globally readable URLs for my Google Cloud Storage objects using the google-cloud-storage Python library (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) - more specifically the…
Matti
  • 299
  • 2
  • 12
9
votes
1 answer

How to authenticate with gcloud big query using a json credentials file?

In the gcloud documentation for google bigquery, it states that authentication can be determined from from_service_account_json. I've tried the following from gcloud import bigquery client =…
Chris
  • 4,194
  • 9
  • 52
  • 100
8
votes
1 answer

Mock the result of accessing public GCS bucket

I have the following code: bucket = get_bucket('bucket-name') blob = bucket.blob(os.path.join(*pieces)) blob.upload_from_string('test') blob.make_public() result = blob.public_url # result is `
Rodrigo
  • 492
  • 7
  • 27
7
votes
2 answers

Retrieve list of log names from Google Cloud Stackdriver API with Python

I'm using Google's Stackdriver Logging Client Libraries for Python to programmatically retrieve log entries, similar to using gcloud beta logging read. Stackdriver also does provide an API to retrieve a list of log names, which is most probably what…
otto.poellath
  • 3,831
  • 6
  • 35
  • 56
7
votes
2 answers

Set metadata in Google Cloud Storage using gcloud-python

I am trying to upload a file to Google Cloud Storage using gcloud-python and set some custom metadata properties. To try this I have created a simple script. import os from gcloud import storage client = storage.Client('super secret app…
Remco Haszing
  • 5,879
  • 3
  • 33
  • 69
6
votes
1 answer

google-cloud-storage distribution not found despite being installed in venv

As in the title, when running the appserver I get a DistributionNotFound exception for google-cloud-storage: File "/home/[me]/Desktop/apollo/lib/pkg_resources/init.py", line 867, in resolve raise DistributionNotFound(req, requirers) …
J Shingle
  • 61
  • 3
6
votes
3 answers

Load schema json file to create table or job schema

If I already have the schema file, for example: schema.json. How can I load the file to create the table or job schema using the google-cloud-python API?
Tuan Vu
  • 708
  • 4
  • 14
6
votes
1 answer

Google Cloud Datastore timeout errors in python

I have a table in Google Cloud Datastore where I store a small data structure which is written in one Python service and read in another. I am using gcloud version 0.15.0. Here is the Python code that I use to write/read data to/from GCD: from…
Amit
  • 61
  • 2
6
votes
2 answers

Difficulty comparing generated and google cloud storage provided CRC32c checksums

I am attemptting to get a CRC32c checksum on my local file so I can compare it to the blob.crc32c provided by the gcloud library. Google says I should be using the crcmod module in order to actually calculate CRC32c hashes of my data.…
Stephen Cagle
  • 12,627
  • 14
  • 49
  • 83
5
votes
2 answers

How to use Google Cloud PubSub and Run to handle resource-intensive long-running tasks?

I've got a Google Cloud PubSub topic which at times has thousands of messages and at times zero messages coming in. These messages represent tasks which can take upwards of an hour each. Preferably I'm able to use Cloud Run for this, as it scales…
thimo visser
  • 75
  • 6
1
2 3
…
12 13