14

Okay, so I messed up, I accidentally ran az ad sp reset-credentials against the Service Principal that our AKS cluster runs under. And now we are getting errors like:

Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}

and

Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required

So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.

Any ideas?

PeterH
  • 681
  • 1
  • 5
  • 16

3 Answers3

12

In the end the solution was quite simple.

  • In the Azure portal, navigate to the resource group named MC_<resourcegroup>_<aksName>_<region>.
  • Click one of the resources of the type "Virtual machine".
  • Scroll down to "Run command"

Run command

  • Choose "RunShellScript"
  • Enter cat /etc/kubernetes/azure.json and click "Run"

The command will return the contents of the JSON file. The property you need is aadClientSecret

PeterH
  • 681
  • 1
  • 5
  • 16
6

Whoever comes over this issue there's an updated solution from Microsoft

https://docs.microsoft.com/en-us/azure/aks/update-credentials#update-aks-cluster-with-new-credentials

They also mention (something that's not obvious) that: By default, AKS clusters are created with a service principal that has a one-year expiration time.

Also, As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords. so the initial solution to change the service principal password doesn't work anymore.

Iulian Paraian
  • 86
  • 1
  • 1
0

It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.

First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:

enter image description here

Or you can use the Azure CLI command to find the registry ID like this:

az acr show --resource-group groupName --name registryName --query id --output tsv

Then use the command to find the service principal ID like this:

az role assignment list --scope registryID

You can select the service principal which you want.

Then you would get all the secrets with the command kubectl get secrets and kubectl get secrets secretName -o yaml to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:

enter image description here

If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.

By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.

Charles Xu
  • 24,004
  • 1
  • 12
  • 27
  • Thanks Charles but I don't think that's going to work. The problem isn't just that the cluster has lost access to ACR. The cluster is using the wrong password to connect to Azure to create loadbalancers, ip addresses, etc. I don't think that password was stored in a K8s secret. According to the docs "On the master and agent VMs in the Kubernetes cluster, the service principal credentials are stored in the file /etc/kubernetes/azure.json". So I think I need to get a hold of that file somehow. – PeterH Aug 24 '18 at 10:29
  • @PeterH The result I posted was what I have tested. You can try it. As you say, the secret store in the file /etc/kubernetes/azure.json. The master and agent are containers and the file /etc/kubernetes/azure.json is inside. If you're right. You can check the secret outside or connect into the container to get the info inside. Which do you think is more convenient. – Charles Xu Aug 27 '18 at 02:02
  • Connecting into the container was my preferred option but how do I connect into the container? – PeterH Aug 27 '18 at 13:28
  • I have tried your method but none of the tokens was in the format that you show. I think that's because I never created the secret as we didn't use ImagePullSecrets but just granted the service principal access to the ACR so the secret wasn't needed. – PeterH Aug 27 '18 at 13:43
  • I found the solution, you can just run a command through the Azure portal. I've provided the solution in an answer. – PeterH Aug 27 '18 at 14:28