Programmatic access from a service account to a Google IAP protected resource denied with invalid signature error

Question

I recently enabled IAP in GKE cluster.

Cluster Version: 1.15.11-gke.11

I followed the instructions here: https://cloud.google.com/iap/docs/enabling-kubernetes-howto

Service config is as follows:

---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: foo-bc-iap
  namespace: foo-test
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: iap-client-secret
---
apiVersion: v1
kind: Service
metadata:
  name: foo-internal-service
  namespace: foo-test
  annotations:
    cloud.google.com/backend-config: '{"ports":{"80":"foo-bc-iap"}}'
spec:
  type: NodePort # To create Ingress using the service.
  selector:
    app: foo-test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8081

The credential I used was OAuth 2.0 Client ID (Type: Web Application).

After making sure the IAP-protected API endpoint works differently when I activate IAP on the Kubernetes service, I wrote the following test program to make sure the endpoint is accessible from the service account given in the JSON file 'account.json'.

In writing this sample application, I consulted this doc: https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-go

google.golang.org/api v0.23.0
go 1.12

func (m *myApp) testAuthz(ctx *cli.Context) error {
    audience := "<The client ID of the credential mentioned above>"
    serviceAccountOption := idtoken.WithCredentialsFile("account.json")

    client, err := idtoken.NewClient(ctx.Context, audience, serviceAccountOption)
    if err != nil {
        return fmt.Errorf("idtoken.NewClient: %v", err)
    }

    requestBody := `{
        <some JSON payload>
    }`

    request, err := http.NewRequest("POST", "https://my.iap.protected/endpoint",
        bytes.NewBuffer([]byte(requestBody)))
    if err != nil {
        return fmt.Errorf("http.NewRequest: %v", err)
    }

    request.Header.Add("Content-Type", "application/json")

    response, err := client.Do(request)
    if err != nil {
        return fmt.Errorf("client.Do: %v", err)
    }
    defer response.Body.Close()

    fmt.Printf("request header = %#v\n", response.Request.Header)
    fmt.Printf("response header = %#v\n", response.Header)

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        return fmt.Errorf("ioutil.ReadAll: %v", err)
    }

    fmt.Printf("%d: %s\n", response.StatusCode, string(body))

    return nil
}

But when I run this, I could only see the following response.

request header = http.Header{"Authorization":[]string{"Bearer <jwt token>"}, "Content-Type":[]string{"application/json"}, "X-Cloud-Trace-Context":[]string{"c855757f20d155da1140fad1508ae3e5/17413578722158830486;o=0"}}

response header = http.Header{"Alt-Svc":[]string{"clear"}, "Content-Length":[]string{"49"}, "Content-Type":[]string{"text/html; charset=UTF-8"}, "Date":[]string{"Wed, 06 May 2020 22:17:43 GMT"}, "X-Goog-Iap-Generated-Response":[]string{"true"}}

401: Invalid IAP credentials: JWT signature is invalid

As you can see here, the access was denied.

So I thought that the signature used to sign the JWT token in the header might be wrong.

But I made sure the following using jwt.io:

The JWT token used in the header is signed by the private key of the caller's service account
The JWT token used in the header can be verified by the public key of the caller's service account
The JWT token was signed using RS256 algorithm

And I also looked into the token:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "<the service account's private key ID>"
}

{
  "iss": "<email address of the service account>",
  "aud": "",
  "exp": 1588806087,
  "iat": 1588802487,
  "sub": "<email address of the service acocunt>"
}

Nothing quite odd.

So I'm not sure what's going on here. If I disable IAP, the endpoint return the right response.

Can anyone give me some hint about what I am doing wrong?

Thanks for the comment! But this is a token created by google library, I couldn't modify it. Further, because the signature verification itself is failing, I doubt that's the root cause. — Byungjoon Lee, May 07 '20 at 16:01
Have you make sure that the service account is at least an IAP web user? — irvifa, May 07 '20 at 19:52

score 3 · Accepted Answer · answered May 16 '20 at 01:42

I tried your code, and I found that it doesn't work with google.golang.org/api v0.23.0, but it does work with google.golang.org/api v0.24.0 (latest one at the time of writing).

It is indeed a bug, the release notes mention the following:

When provided, use the TokenSource from options for NewTransport. This fixes a bug in idtoken.NewClient where the wrong TokenSource was being used for authentication.

Interestingly, 0.23.0 is sending a token signed with the service account's private key with the following claims:

{
  "iss":"xx@xx.iam.gserviceaccount.com",
  "aud":"",
  "exp":1589596554,
  "iat":1589592954,
  "sub":"xx@xx.iam.gserviceaccount.com"
}

and 0.24.0 sends a token signed with google's private key. (internally the previous token is exchanged for a google-signed token)

{
    "aud":"xxxxxxx-xxxxxxxxxxxxxxxxxx.apps.googleusercontent.com",
    "azp":"xx@xx.iam.gserviceaccount.com",
    "email":"xx@xx.iam.gserviceaccount.com",
    "email_verified":true,
    "exp":1589596508,
    "iat":1589592908,
    "iss":"https://accounts.google.com",
    "sub":"11524xxxxxxxxxxxxxxxx"
}

Thanks @Dirbaio! I also agree that it's a bug. I will add an updated code as a workaround. — Byungjoon Lee, May 28 '20 at 02:40

score 0 · Answer 2 · answered May 15 '20 at 09:33

0

JWT token exp property seems to have an expired date (7th May 1:01) and your post was from (7th May 3:53) have you tried regenerating it?

[EDIT: I know it should be a comment. Posted as answer because of low rep]

answered May 15 '20 at 09:33

Gosia B.

11
3

Thanks for the comment! but it was not relevant to the issue. – Byungjoon Lee May 28 '20 at 02:39

score 0 · Answer 3 · answered May 28 '20 at 02:52

As @Dirbaio pointed out, I think this is an issue specific to v0.23.0. Because I cannot upgrade the dependency right now, I chose to create a new IAP client that does not use idtoken.NewClient. Instead, it just uses idtoken.NewTokenSource to create an OIDC token. Adding the token to the Authorization header is easy so I could work around the issue in the client created by idtoken.NewClient.

package main

import (
    "context"
    "crypto/tls"
    "fmt"
    "io"
    "net/http"

    "golang.org/x/oauth2"
    "google.golang.org/api/idtoken"
    "google.golang.org/api/option"
)

// IAPClient is the default HTTPS client with Morse-Code KMS integration.
type IAPClient struct {
    client      *http.Client
    tokenSource oauth2.TokenSource
}

// NewIAPClient returns an HTTP client with TLS transport, but not doing the CA checks.
func NewIAPClient(audience string, opts ...option.ClientOption) *IAPClient {
    tokenSource, err := idtoken.NewTokenSource(context.Background(), audience, opts...)
    if err != nil {
        panic(fmt.Errorf("cannot create a new token source: %s", err.Error()))
    }

    return &IAPClient{
        client: &http.Client{
            Transport: &http.Transport{
                TLSClientConfig: &tls.Config{
                    InsecureSkipVerify: true,
                },
            },
        },
        tokenSource: tokenSource,
    }
}

// Do sends the http request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Do(request *http.Request) (*http.Response, error) {
    err := c.addAuthorizationHeader(request)
    if err != nil {
        return nil, fmt.Errorf("couldn't override the request with the new auth header: %s", err.Error())
    }

    return c.client.Do(request)
}

// Get sends the http GET request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Get(url string) (*http.Response, error) {
    request, err := http.NewRequest(http.MethodGet, url, nil)

    if err != nil {
        return nil, err
    }

    return c.Do(request)
}

// Post sends the http POST request to server with a morse-code JWT Authorization: Bearer header.
func (c *IAPClient) Post(url, contentType string, body io.Reader) (*http.Response, error) {
    request, err := http.NewRequest(http.MethodPost, url, body)
    if err != nil {
        return nil, err
    }

    request.Header.Add("Content-Type", contentType)

    return c.Do(request)
}

func (c *IAPClient) addAuthorizationHeader(request *http.Request) error {
    tkn, err := c.tokenSource.Token()
    if err != nil {
        return fmt.Errorf("cannot create a token: %s", err.Error())
    }

    tkn.SetAuthHeader(request)

    return nil
}

Programmatic access from a service account to a Google IAP protected resource denied with invalid signature error

3 Answers3