GCP Authentication: RefreshError

Question

In order to round-trip test mail sending code in our GCP backend I am sending an email to a GMail inbox and attempting to verify its arrival. The current mechanism for authentication to the GMail API is fairly standard, pasted from the GMail API documentation and embedded in a function:

def authenticate():
    """Authenticates to the Gmail API using data in credentials.json,
    returning the service instance for use in queries etc."""
    store = file.Storage('token.json')
    creds = store.get()
    if not creds or creds.invalid:
        flow = client.flow_from_clientsecrets(CRED_FILE_PATH, SCOPES)
        creds = tools.run_flow(flow, store)
    service = build('gmail', 'v1', http=creds.authorize(Http()))
    return service

CRED_FILE_PATH points to a downloaded credentials file for the service. The absence of the token.json file triggers its re-creation after an authentication interaction via a browser window, as does the token's expiry.

This is an integration test that must run headless (i.e. with no interaction whatsoever). When re-authentication is required the test currently raises an exception when the authentication flow starts to access sys.argv, which means it sees the arguments to pytest!

I've been trying to find out how to authenticate reliably using a mechanism that does not require user interaction (such as an API key). Nothing in the documentation or on Stackoverflow seems to answer this question.

A more recent effort uses the keyfile from a service account with GMail delegation to avoid the interactive Oauth2 flows.

def authenticate():
    """Authenticates to the Gmail API using data in g_suite_access.json,
    returning the service instance for use in queries etc."""
    main_cred = service_account.Credentials.from_service_account_file(
        CRED_FILE_PATH, scopes=SCOPES)
    # Establish limited credential to minimise any damage.
    credentials = main_cred.with_subject(GMAIL_USER)
    service = build('gmail', 'v1', credentials=credentials)
    return service

On trying to use this service with

        response = service.users().messages().list(userId='me',
                                    q=f'subject:{subject}').execute()

I get:

google.auth.exceptions.RefreshError:
  ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.',
   '{\n "error": "unauthorized_client",\n "error_description": "Client is unauthorized to retrieve access tokens using this method."\n}')

I get the feeling there's something fundamental I'm not understanding.

Why can not you use IMAP access to the gmail account to perform this integration test? I understand that this is not what you wanted to get help with, but this seems to be much simpler to achieve the goal of integration test as described. — isp-zax, Sep 09 '18 at 02:55
I did consider that, but since we'll want to use Gmail API services for other purposes later this would just be kicking the problem down the road. — holdenweb, Sep 09 '18 at 11:48
Ironically, investigating the IMAP solution further, I had trouble authenticating with IMAP, and further research led me to [this question](https://stackoverflow.com/questions/25413301/gmail-login-failure-using-python-and-imaplib), where the answer is ... to use OAuth2 authentication!! — holdenweb, Sep 09 '18 at 18:47
Well, that is only if you disable "access for less secure apps" as it says there. So technically it's not a stopper for checking that e-mail arrives - just need to change this setting to allow. But, since you need GMail API later anyway, that's clearly not a permanent solution, just possibly a temporary workaround. IMAP totally works though - I move gmail inboxes contents elsewhere using it. — isp-zax, Sep 09 '18 at 22:41
Yes, I now have an IMAP fallback solution, but we've agreed for the moment this can be a manually-run integration test. We don't need to check our email channels every minute, but it wold be nice to know everything worked after a new vendor release, for example. I'm still hoping the bounty drags in an answer, but I'm not holding my breath. — holdenweb, Sep 10 '18 at 20:50
[Maybe this ?](https://stackoverflow.com/questions/26736062/sending-email-fails-when-two-factor-authentication-is-on-for-gmail) — dsgdfg, Sep 11 '18 at 06:28
'Fraid not, but all such questions welcome. I've now abandoned the IMAP solution in favour of `flow_from_client_secrets` - if we are running interactively the tester can assert the right authentication credentials. — holdenweb, Sep 11 '18 at 11:57
The gmail id that you are trying with, is it a gsuite id or a simple gmail id(@gmail.com)? — Shubham Sinha, Sep 12 '18 at 13:02
It's G-suite (our own domain - one error was to try and login in under an aliased domain, which won't work). — holdenweb, Sep 12 '18 at 22:01
I tried running your code and it's works absolutely fine. Please check two things - 1. If you are authorising the correct scopes. 2. I've generally seen this error come when scopes are missing from service account authorization. Have you followed this correctly - https://developers.google.com/admin-sdk/directory/v1/guides/delegation ? — Shubham Sinha, Sep 14 '18 at 15:19
Yes, very useful. As you suspected, the service account authorization wasn't correctly scoped. — holdenweb, Sep 16 '18 at 15:17
I believe it's necessary to write an answer to claim a bounty, and there was only one answer when the bounty was up. Sorry! — holdenweb, Sep 18 '18 at 03:16

score 3 · Accepted Answer · answered Sep 15 '18 at 05:52

The service account needs to be authorized or it cant access the emails for the domain.

"Client is unauthorized to retrieve access tokens using this method"

Means that you have not authorized it properly; check Delegating domain-wide authority to the service account

Source: Client is unauthorized to retrieve access tokens using this method Gmail API C#

GCP Authentication: RefreshError

1 Answers1

Linked