14

I written a script that extract some data from an API and build an Excel file. I'm not a dev, it is my first real program ever writted. I hosted the code on Google Colab.

There is API secret keys in clear. I want to share it with a Google Drive sharing link to people needing to generate the Excel file so that they can execute it. However I would prefer not to include API secret keys in clear in order to avoid accidental sharings outside of the entreprise.

I'm wondering how to hide this... Or how to provide users an alternative methode to execute the file without knowing the passwords. I don't have access to a shared webserver internally to the entreprise.

Regards

CLIENT_KEY = u'*****'
CLIENT_SECRET = u'*****'
BASE_URL = u'*****'

access_token_key = '*****'
access_token_secret = '*****'


print ('Getting user profile...',)
oauth = OAuth(CLIENT_KEY, client_secret=CLIENT_SECRET, resource_owner_key=access_token_key,
              resource_owner_secret=access_token_secret)
r = requests.get(url=BASE_URL + '1/user/me/profile', auth=oauth)
print (json.dumps(r.json(), sort_keys=True, indent=4, separators=(',', ': ')))

...
Xiiryo
  • 1,361
  • 1
  • 9
  • 31
  • Sounds like you want to set an environment variable. I haven't used Google Colaboratory before, but I would look for a feature that allows you to set an environment variable. Then, in your code you can `import os` and use `os.environ["CLIENT_KEY"]` and so on. – Jack Moody Feb 07 '19 at 11:02
  • It seems to be intersting but since Colab is based on a temporary environment, wouldn't it mean to generate the environnement variable each time, wich will make the keys available somewhere else in the code ? – Xiiryo Feb 07 '19 at 17:13
  • It is fairly ridiculous that you can't set Colab environment variables at the account level. In many ways Colab is a toy. – rjurney Aug 24 '20 at 19:43

4 Answers4

10

Try getpass. For example:

from getpass import getpass
secret = getpass('Enter the secret value: ')

Then, you can share the notebook and each user can enter a distinct value, which you can then use later in the notebook as a regular Python variable.

Bob Smith
  • 26,929
  • 9
  • 72
  • 69
  • Thanks. I feel lke giving 4 keys to copy past will be overhelmingly too much complicated to some. But I could find a way to encrypt the 4 keys in the code and make it avaible only with a password. – Xiiryo Feb 07 '19 at 17:10
  • works great. just remember to not print the variable later – s2t2 Dec 19 '20 at 17:34
7

You can save the secret key as file on Google Drive. Then read the file into Colab.

Now you can set permission to access the key file in Google Drive. Only you and the people you share the key file can use it.

Update

As @efbbrown suggest, you can create an aws key file and store it in Google Drive, e.g.

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

But now (2020) you don't need pydrive any more. You can just

  • Open the file pane on the left side of Colab.
  • Select 'Mount Drive'
  • Accept by clicking 'Connect to Google Drive'
  • Copy that file to Colab, using the code below.

Default place to store credential is ~/.aws/config. So you can do this (if your file above is named aws_config)

!mkdir -p ~/.aws
!cp "/content/drive/My Drive/aws_config" ~/.aws/config
korakot
  • 24,489
  • 13
  • 84
  • 114
  • As I commented above, this requires an interactive authentication with Google Drive, which is not desirable for many cases. To some extent this kicks the can down the road but does mitigate the problem by using Google authentication instead of a shared secret. Now though you've got to maintain a shared drive with the credentials on it and grant permission to that shared drive to each user. – rjurney Aug 24 '20 at 19:42
  • @rjurney If you create a new colab notebook, you can connect it with google drive automatically (with UI, no need to copy & paste token). – korakot Aug 25 '20 at 02:05
7

To expand on @Korakot Chaovavanich's answer, here is the step by step of that solution:

  1. Create a file and save it to google drive with your keys in it. It should look like this:
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  1. Install pydrive
!pip install -U -q PyDrive
  1. Authenticate for google drive, download & parse the creds file

(Some of this code comes from @wenkesj's answer on this question.)

# Imports
import os
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Google drive authentication
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# File params
local_save_dir = "/root/.aws"
filename = "credentials"
save_path = "{0}/{1}".format(local_save_dir, filename)

# Choose/create a local (colab) directory to store the data.
local_download_path = os.path.expanduser(local_save_dir)
try:
  os.makedirs(local_download_path)
except: pass

drive_list = drive.ListFile().GetList()
f = [x for x in drive_list if x["title"] == filename][0]

print('title: %s, id: %s' % (f['title'], f['id']))
fname = os.path.join(local_download_path, f['title'])
print('downloading to {}'.format(fname))
f_ = drive.CreateFile({'id': f['id']})
f_.GetContentFile(fname)

with open(save_path) as creds:
    for i, line in enumerate(creds):
        if i == 1:
            access_token_key = line.replace("aws_access_key_id=", "").replace("\n", "")
        if i == 2:
            access_token_secret = line.replace("aws_secret_access_key=", "").replace("\n", "")

Now your AWS keys are in the two variables access_token_key & access_token_secret.

efbbrown
  • 3,113
  • 5
  • 26
  • 45
  • To some extent you've just used Google authentication to kick the can down the road. The user must authenticate into Google Drive for this to work, now the burden is on authenticating with that service - which is interactive and involves pasting a token. That is highly undesirable, but if it works for you - great! – rjurney Aug 24 '20 at 19:41
  • @rjurney do you have an alternative solution that allows the user to avoid embedding their keys in the notebook? – efbbrown Aug 25 '20 at 09:06
  • @effbrown I use S3. I copy the files there and then download them in the Colab notebook using boto3 in 3 LOC. This way I can be certain the files have been uploaded and downloaded rather than waiting for Google Drive to sync. Sometimes hundreds of other files would be higher in the queue for syncing and I could not run my latest code. S3 fixes this. For the moment on a project I put the credentials in the notebook but in general I prompt the user for credentials with https://pypi.org/project/get-input/ or https://docs.python.org/3/library/getpass.html in the same way Google Drive does. – rjurney Aug 27 '20 at 22:44
2

I would recommand using GCP's Secret Manager :

You get useful features such as rights management (in IAM & Admin), you can update your passwords via Secret versions, etc.. really useful.

Pre-requisits:

  • Have a Google Cloud Platform project
  • In the IAM & Admin, you should have the role : "Secret Manager Secret Accessor"
  • Create a Secret in the Secret Manager :

Here is a way to get your secret with python 3 :

# Install the module and import it :
!pip install google-cloud-secret-manager
from google.cloud import secretmanager

# Create a Client:
client = secretmanager.SecretManagerServiceClient()
secret_name = "my-secret" # => To be replaced with your secret name
project_id = 'my-project' # => To be replaced with your GCP Project

# Forge the path to the latest version of your secret with an F-string:
resource_name = f"projects/{project_id}/secrets/{secret_name}/versions/latest" 

# Get your secret :
response = client.access_secret_version(request={"name": resource_name})
secret_string = response.payload.data.decode('UTF-8')

# Tada ! you secret is in the secret_string variable!

Do not try it with your real password or secret while testing this.

Enjoy !

yudhiesh
  • 3,560
  • 3
  • 7
  • 24
Geoffroy de Viaris
  • 61
  • 4