KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

Question

I'm trying to set up my web app, hosted in Azure to read settings from Azure KeyVault.

I've been following this guide: https://anthonychu.ca/post/secrets-aspnet-core-key-vault-msi/

The example shows how to access app settings from KeyVault with the configuration:

public static IWebHost BuildWebHost(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
     .ConfigureAppConfiguration((ctx, builder) =>
     {
         var keyVaultEndpoint = Environment.GetEnvironmentVariable("KEYVAULT_ENDPOINT");
         if (!string.IsNullOrEmpty(keyVaultEndpoint))
         {
             var azureServiceTokenProvider = new AzureServiceTokenProvider();
             var keyVaultClient = new KeyVaultClient(
                 new KeyVaultClient.AuthenticationCallback(
                     azureServiceTokenProvider.KeyVaultTokenCallback));
             builder.AddAzureKeyVault(
                 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
         }
     })
    .UseApplicationInsights()
    .UseStartup<Startup>()
    .Build();

I've added the KEYVAULT_ENDPOINT environment variable to the application settings. I've enabled MSI on the app service, and I've authorized my Azure User, and my application, from the Key Vault Access Policies:

With Get and List Operations:

And I've added the secret to key vault. Running locally, I can access the secret.

But my ASP .NET Core site fails on startup with this in the stdout logs:

Unhandled Exception: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Blog.Program.BuildWebHost(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 22
   at [csproj name].Program.Main(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 16

I've checked that MSI_ENDPOINT and MSI_SECRET environment variables exist by calling SET from the debug console.

I can also see the KEYVAULT_ENDPOINT variable.

Any suggestions what could be going wrong or on what to try next? Since it works locally it must be an authentication issue, but I believe it's authenticating locally with my azure User that I've authorized in key vault, rather than as the Azure App Service.

It's off topic here but remember that KeyVaultClient is IDisposable so it would be better to enclosure it with "using" — Andrzej Martyna, Aug 10 '18 at 08:26

score 34 · Accepted Answer · answered Sep 10 '18 at 05:56

It is dodgy, but the problem with your setup is that you enabled "Authorized application" when you created the Access policy for your KeyValut.

You can see it because you have "Application+Application" on your screenshot. I imagine, you created the policy with your webapp being both principal and authrorized application. This kinda creates a loop.

To make this setup work just delete your existing policy and create a new one, where you select only the Principal:

This is the right answer, everything is working fine. – Ashish-BeJovial May 04 '21 at 10:59 — Ashish-BeJovial, May 04 '21 at 10:59

score 3 · Answer 2 · answered Sep 13 '18 at 14:49

3

only select principal and add web app,don't select authorized application

answered Sep 13 '18 at 14:49

Viral Jain

56
3

Mark Szabo · Answer 3 · 2020-07-18T11:33:08.703

3

For me to resolve the issue, I had to add the IP addresses of the web app to the firewall as not all of them were getting added when you select "Allow trusted Microsoft services..."

Thanks to: https://azidentity.azurewebsites.net/post/2019/01/03/key-vault-firewall-access-by-azure-app-services

edited Jul 18 '20 at 11:33

answered Jan 26 '19 at 16:15

Mark Szabo

4,023
3
16
25

score 2 · Answer 4 · answered Apr 02 '19 at 09:45

Check to ensure that the Service Principal you are using has been added and has the appropriate permissions for the operations you want to perform (Keys/Secrets etc):

Check to see if the KeyVault has restricted network access, toggle to test if this is the issue or not:

Svek · Answer 5 · 2018-07-03T03:35:24.053

I have seen issues like this before, with development and production systems not behaving the same way when dealing with environment variables.

My first area to look at in these scenarios is to check if you have properly prefixed your environment variables with ASPNETCORE_ in your production system. This prefix is the default prefix for ASP.NET Core web hosts.

| Key                                |  Value                     |
|====================================|============================|
| ASPNETCORE_KEYVAULT_ENDPOINT       |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_ENDPOINT**          |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_SECRET**            |  xxxxxxxxxxxxxxxx          |

** Note that in your code I don't ever see you using MSI_ENDPOINT or MSI_SECRET. I only see you using KEYVAULT_ENDPOINT

If you want to explicitly set the prefix to something else, the prefix you want to use can be passed as an argument when initializing the web host via the configuration.

.AddEnvironmentVariables("ASPNETCORE_"); // choose your own prefix here

Thanks, what's the purpose of prefixing with ASPNETCORE_? MSI_ENDPOINT and MSI_SECRET are read by the Microsoft.Azure.KeyVault package itself to do MSI authentication (I think). — zola25, Jul 02 '18 at 18:32
@zola25 I wasn't sure about those two keys, as they were not defined in your question. However, the ASPNETCORE_ prefix is the default prefix for ASP.NET Core web hosts unless you explicitly change that. — Svek, Jul 03 '18 at 03:32

score 1 · Answer 6 · answered Jul 17 '20 at 16:59

1

In the AZ KV, access policies... by selecting only the principal should work.

enter image description here

answered Jul 17 '20 at 16:59

Carlos Cruz

105
1
5

score 1 · Answer 7 · answered Apr 28 '21 at 17:30

I want to add another case here with the same symptoms/error messages. Azure keyvault returned Forbidden when one of the secrets was disabled.

I had all the required permissions - I was accessing keyvault with Get and List granted for secrets (I was using only secrets) for the Managed Identity of my App Service.

It was confusing to get the "Access Forbidden" error, and it took some experimentation to figure out the issue. Hope this helps someone who have faced the same issue.

score 0 · Answer 8 · answered Apr 16 '21 at 17:00

Another possibility is that the service principal needs API permissions for Key Vault. In Azure portal, go to Service Principal, then choose API permissions on the left hand menu. Then click Add permissions. Here you can choose Key Vault and save. This will allow the code to retrieve the key vault secrets.

KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

8 Answers8

Linked