How to search for Tensorflow Files in Google Drive?

Question

I'm following the docs here: https://colab.research.google.com/github/google/earthengine-api/blob/master/python/examples/ipynb/TF_demo1_keras.ipynb#scrollTo=43-c0JNFI_m6 to learn how to use Tensorflow with GEE. One part of this tutorial is checking the existence of exported files. In the docs, the example code is:

fileNameSuffix = '.tfrecord.gz'
trainFilePath = 'gs://' + outputBucket + '/' + trainFilePrefix + fileNameSuffix
testFilePath = 'gs://' + outputBucket + '/' + testFilePrefix + fileNameSuffix

print('Found training file.' if tf.gfile.Exists(trainFilePath) 
    else 'No training file found.')
print('Found testing file.' if tf.gfile.Exists(testFilePath) 
    else 'No testing file found.')

In my case, I'm just exporting the files to Google Drive instead of Google Cloud bucket. How would I change trainFilePath and testFilePath to point to the Google Drive folder? FWIW, when I go into the Google Drive Folder, I do actually see files.

Nebulastic · Answer 1 · 2020-02-25T21:52:14.900

I would say you could use the Google Drive API to list files in you Google Drive instead of a GCS Bucket. You can find the documentation here.

You can also use PyDrive, which is pretty easy to understand. This is an example, you only have to adjust the query "q" to your needs:

from pydrive.drive import GoogleDrive
from pydrive.auth import GoogleAuth

gauth = GoogleAuth()

gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)

file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file in file_list:
  print(f"title: {file['title']}, id: {file['id']}")

score 0 · Answer 2 · edited Mar 18 '20 at 14:28

Solution

You can use the great PyDrive library to acess your Drive files easily from Google collab and thus check which files you have or have been exported, etc.

The following piece of code is an example that lists all the files in the root directory of your Google Drive API. This has been found in this answer (yes, I am making this answer a community wiki post) :

# Install the library
!pip install -U -q PyDrive
# Install the rest of the services/libraries needed
import os
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# choose a local (colab) directory to store the data.
local_download_path = os.path.expanduser('~/data')
try:
  os.makedirs(local_download_path)
except: pass

# 2. Auto-iterate using the query syntax, in this case as I am using the main directory of Drive this would be root
#    https://developers.google.com/drive/v2/web/search-parameters
file_list = drive.ListFile(
    {'q': "'root' in parents"}).GetList()

for f in file_list:
  # 3. Print the name and id of the files
  print('title: %s, id: %s' % (f['title'], f['id']))

NOTE: when you do this colab will take you to another page to authentificate and make you insert the secret key. Just follow what the service indicates you to do, it is pretty straightforward.

I hope this has helped you. Let me know if you need anything else or if you did not understood something. :)

How to search for Tensorflow Files in Google Drive?

2 Answers2

Solution