Access KeyVault from Azure Kubernetes Service (AKS) with an ASP.NET Core application using a Managed Identity

Secrets in Kubernetes are not really secret. You should store all your cloud-related secrets in a vault anyway, so why not access this vault from your Kubernetes cluster. If you are using Azure, Azure KeyVault is the most logical place to store your secrets. This blogpost tells you how to access the KeyVault from an ASP.NET Core application running on AKS.

We are going to use project Aad-Pod-Identity to access the KeyVault from a Pod. Aad-Pod-Identity is an open-source project that Microsoft is offering and contains the needed components to read the KeyVault using a Managed Identity.

It’s also possible to use aad-pod-identity using a Service Principal, but then you still have to store the password somewhere in Kubernetes, which doesn’t make sense. The same story applies to access the KeyVault using a Service Principal direct from the ASP.NET Core application (without aad-pod-identity). You still need to store the password somewhere. So let’s use a Managed Identity.

Steps to do it yourself

I’ve provided a simple sample application that is available on Docker Hub. To use this Docker Image and running it in your AKS cluster it will be able to read secrets from your KeyVault using a Managed Identity.

(You can of course also use your own application and only use the provision.sh script. Then you should make the needed changes to keyvaultreader.yaml.)

1. Get the needed files. Execute:

   git clone https://github.com/pascalnaber/keyvault-access-aks.git

2. Change the blue parameters in the provision.sh script (see below)
3. Change the blue parameter in keyvaultreader.yaml (see below)
4. Run

   ./provision.sh

The text below explains what is happening:

Configure and Provision Resources on Azure

I’m using the following idempotent bash script (provisioning.sh) to:

• create the Managed Identity including all access rights needed
  (in the aks resourcegroup)
• create or use the KeyVault.
• Install aad-pod-identity using Helm 3

#!/bin/bash
set -xe

RESOURCEGROUP_AKS=matrix-k8s-rbac-dev-we
AKS_NAME=matrix-aks-rbac-dev-we
RESOURCEGROUP_KEYVAULT=matrix-keyvault-rbac-dev-we
KEYVAULT_NAME=matrix-secrets-rbac-dev
IDENTITY_AKSKEYVAULT_NAME=identity-kv-rbac-aks-dev
K8S_KEYVAULTACCESS_LABEL=keyvaultaccess

IDENTITY_CLIENTID=$(az identity create -g $RESOURCEGROUP_AKS -n $IDENTITY_AKSKEYVAULT_NAME --query clientId -o tsv)
IDENTITY_RESOURCEID=$(az identity show -g $RESOURCEGROUP_AKS -n $IDENTITY_AKSKEYVAULT_NAME --query id -o tsv)
IDENTITY_PRINCIPALID=$(az identity show -g $RESOURCEGROUP_AKS -n $IDENTITY_AKSKEYVAULT_NAME --query principalId -o tsv)

az keyvault create --name $KEYVAULT_NAME --resource-group $RESOURCEGROUP_KEYVAULT --enabled-for-template-deployment --enabled-for-deployment
sleep 30 # sometimes the Service Principal is not there yet
az keyvault set-policy -n $KEYVAULT_NAME --secret-permissions get list --spn $IDENTITY_CLIENTID

SUBSCRIPTIONID=$(az account show --query id -o tsv)
NODERESOURCEGROUP=$(az aks show --resource-group $RESOURCEGROUP_AKS --name $AKS_NAME --query nodeResourceGroup -o tsv)
az role assignment create --role Reader --assignee $IDENTITY_PRINCIPALID --scope /subscriptions/$SUBSCRIPTIONID/resourcegroups/$NODERESOURCEGROUP

# according to https://github.com/Azure/aad-pod-identity. But not needed at all.
AKS_SERVICEPRINCIPAL_ID=$(az aks show -g $RESOURCEGROUP_AKS -n $AKS_NAME --query servicePrincipalProfile.clientId -o tsv)
az role assignment create --role "Managed Identity Operator" --assignee $AKS_SERVICEPRINCIPAL_ID --scope $IDENTITY_RESOURCEID

# The Helm charts gives an error. It's missing the CRD's. Install them manually to make it work.
# More info: https://github.com/Azure/aad-pod-identity/issues/454
kubectl apply -f ./aadpodidentity-tmp/crd.yaml
sleep 120 # sometimes the Service Principal is not available for AKS
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm upgrade "aad-pod-identity" "aad-pod-identity/aad-pod-identity" --install --set azureIdentity.enabled=true,azureIdentity.resourceID=$IDENTITY_RESOURCEID,azureIdentity.clientID=$IDENTITY_CLIENTID,azureIdentityBinding.selector=$K8S_KEYVAULTACCESS_LABEL

kubectl apply -f keyvaultreader.yaml

The only thing you have to modify is the first five variables (in blue) with your situation.

In case of an error

Especially the 2nd sleep is important. Sometimes, when there is a new Managed Identity, aad-pod-identity is not able to use the Managed Identity. The following exception comes up when that happens (in one of the nmi pods):

# time="2020-01-20T22:02:50Z" level=error msg="failed to get service principal token for pod:default/keyvaultreader-f4cfb6467-pwddl, adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=10.240.0.135

Have a look at the logs of the pods:

• aad-pod-identity-mic
• aad-pod-identity-nmi

Waiting is not fixing it. What works for me is to delete the helm repo again and the application. And just install it again by running the script above another time.

helm delete "aad-pod-identity" 
kubectl delete -f keyvaultreader.yaml

The .NET Core sample application

The Docker image being used in keyvaultreader.yaml is a sample .NET Core 3 application that reads a secret from the KeyVault using aad-pod-identity. So using the Managed Identity.

The yaml looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: keyvaultreader
    aadpodidbinding:keyvaultaccess
  name: keyvaultreader  
spec:
  template:
    metadata:
      labels:
        app: keyvaultreader
        aadpodidbinding:keyvaultaccess
    spec:
      containers:
      - name: keyvaultreader
        image: "pascalnaber/keyvaultreader:1"
        env:        
        - name: KeyVaultName
          value: "matrix-secrets-rbac-dev"
        imagePullPolicy: Always
        ports:
          - containerPort: 80

Both scripts must have a matching label. To make it clear for you, the font color is yellow in both scripts.

The Deployment contains one environment variable. The value should be the name of the KeyVault.

In Visual Studio

Add two NuGet packages:

• Microsoft.Extensions.Configuration.AzureKeyVault
• Microsoft.Azure.Services.AppAuthentication

program.cs look like this:

 public static IHostBuilder CreateHostBuilder(string[] args) =>
 
        Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, config) =>
            {
                var builtConfig = config.Build();
 
                if (string.IsNullOrEmpty(builtConfig["KeyVaultName"]))
                    throw new Exception("Configure the KeyVaultName by using an Environment Variable");
 
                var azureServiceTokenProvider = new AzureServiceTokenProvider();
                var keyVaultClient = new KeyVaultClient(
                    new KeyVaultClient.AuthenticationCallback(
                        azureServiceTokenProvider.KeyVaultTokenCallback));
 
                config.AddAzureKeyVault(
                    $"https://{builtConfig["KeyVaultName"]}.vault.azure.net/",
                    keyVaultClient,
                    new DefaultKeyVaultSecretManager());
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

Now you are able to read the secrets as you are used to reading configuration values:

public class HomeController : Controller
{
    private readonly ILogger<HomeController> _logger;
    private readonly IConfiguration _config;
 
    public HomeController(ILogger<HomeController> logger, IConfiguration config)
    {
        _logger = logger;
        _config = config;
    }
 
    public IActionResult Index(SecretsViewModel viewModel)
    {
        viewModel.Secret = _config.GetValue<string>(viewModel.SecretName);
        return View(viewModel);
    }

 

Conclusion

With the provided scripts, configuration files and .NET Core 3 application, it should be easy to access the KeyVault from your ASP.NET Core application using a Managed Identity. We are making use of aad-pod-identity. A handy project, but doesn’t feel very mature yet. Sometimes exceptions come up during setup, which can only be solved by deleting and provisioning again. Also the Helm Chart is not what I’m expecting how it should work.

All sources can be found here:

https://github.com/pascalnaber/keyvault-access-aks

The sample application is available to use from Docker also:

https://hub.docker.com/repository/docker/pascalnaber/keyvaultreader

Advertentie