Handling settings and Environment Variables of your .NET Core 2 application hosted in a Docker container during development and on Kubernetes (Helm to the resque)

You have developed a microservice in .NET Core 2 and want to host it as a Docker Container in Kubernetes. Your Microservice contains settings, some appsettings or connectionstrings for example. These settings differ over environments. You can treat this configuration for Kubernetes on different ways. This blogposts shows you how to handle settings over environments prepared for Continuous Delivery.

In this blogpost, you can read how to create a Docker container that contains your .NET Core 2 application and host it on Kubernetes.

Let’s assume your .NET Core 2 application contains the following configuration in appsettings.json:

{
  "ServiceBus": {
    "Url": "myServiceBus.servicebus.windows.net",
    "Key": "abcdefghijklmnopqrstuvwzyz"
  },
  "Database": {
    "ConnectionString": "mySqlConnectionstring"
  },
  "Logging": {
    "IncludeScopes": false,
    "Debug": {
      "LogLevel": {
        "Default": "Warning"
      }
    },
    "Console": {
      "LogLevel": {
        "Default": "Warning"
      }
    }
  }
}

There are multiple ways you can handle configuration for multiple environments using a CI/CD pipeline like in VSTS.

1. Environment specific settings files

.NET Core 2 supports a settings file per environment you want to deploy to. Like:

The consequences of putting settings in these files is that all your settings, including passwords, are stored in the docker image. You can override all settings when deploying the image, but the original settings are still in the Docker image and can be seen by everyone who has access to the Docker image. You can also use placeholders in the environment specific appsetting files, but this solution has the same disadvantage when the CI/CD pipeline has replaced the placeholders with the real values: the secret settings are stored in the Docker Image.

Settings for logging, for example, are a good candidate of a setting that could be part of the appsettings.

When you want to use secret settings and don’t want the Docker Image to contain them, you could use a environment specific appsettings file that is not part of the Docker Image:

2. Environment specific settings file attached as a Volume from a secret in Kubernetes

You could add, for example an appsettings.docker.json file as a template for settings. Only a template, so no real values are stored in the settings file. Only placeholders.
The release in your CI/CD pipeline could replace the placeholders with the real values. This appsettings.docker.json file looks like this:

{
  "ServiceBus": {
    "Url": "#{ServiceBusUrl}#",
    "Key": "#{ServiceBusKey}#"
  },
  "Database": {
    "ConnectionString": "#{DatabaseConnectionString}#"
  }
}

This appsettings.docker.json file should be loaded in Program.cs as optional file:

public static IWebHost BuildWebHost(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((buildercontext, config) =>
                {
                    config.AddJsonFile("settings/appsettings.docker.json", optional: true);
                })
                .UseStartup()
                .Build();

The settings folder specified above will be provided by mapping a Volume to the Docker container. First you need to add the secret:

kubectl create secret generic settingsdemo-docker-appsettings --from-file=./appsettings.docker.json

The following deploymentfile deploys a Docker Image and mounts a volume to the secret file. The BuildWebHost mentioned above is reading from this path:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: settingsdemo-deployment
spec:
  replicas: 3 
  template:
    metadata:
      labels:
        app: settingsdemo
    spec:
      containers:
      - name: settingsdemo
        image: myregistry.azurecr.io/settingsdemo:#{ContainerVersion}#          
        ports:
        - containerPort: 80
        imagePullPolicy: Always        
        volumeMounts:
        - name: settings
          mountPath: /app/settings
          readOnly: true
      volumes:
      - name: settings
        secret:
          secretName: settingsdemo-docker-appsettings

This works nice. The Docker Image doesn’t contain secret values. It’s easy to apply this in the CI/CD pipeline. There are other possiblities also. Settings can be specified with Environment variables within the deployment file:

3. Environment Variables in the Kubernetes Deployment file

The deployment file can contain Environment variables also. It’s not needed to have a setting in your setting file to also pass it as a Environment variable. You can read the Environment variable the same way as you read your configuration setting. (see the ExtraSettingNotInSettingsFile Environment variable in the sample deploymentfile below)

public string GetExtraValue(IConfiguration configuration)
{
    return configuration.GetValue<string>("ExtraSettingNotInSettingsFile");
}

If you want to override a setting in .NET Core 2, you can just use the name. If a setting is a child in json, then use the double underscore (__) as seperator. The value can also be a placeholder, which you replace in your CI/CD pipeline.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: settingsdemo-deployment
spec:
  replicas: 3 
  template:
    metadata:
      labels:
        app: settingsdemo
    spec:
      containers:
      - name: settingsdemo
        image: myregistry.azurecr.io/settingsdemo:#{ContainerVersion}#  
	env:
	- name: ServiceBus__Url
          value: "myServiceBus.servicebus.windows.net"  
        - name: ServiceBus__Key
          value: "abcdefghijklmnopqrstuvwzyz"
	- name: Database__ConnectionString
          value: "mySqlConnectionstring"
	- name: ExtraSettingNotInSettingsFile
          value: "extraValue"
        ports:
        - containerPort: 80
        imagePullPolicy: Always        

Remember the settings look like this in the .NET Core 2 app:

"ServiceBus": {
  "Url": "myServiceBus.servicebus.windows.net",
  "Key": "abcdefghijklmnopqrstuvwzyz"
},
"Database": {
  "ConnectionString": "mySqlConnectionstring"
},

With this solution the artifact to deploy to Kubernetes is mixed with variables that change when deploying to different environments. There is another possibility that can solve this:

4. Use Helm for deployment to Kubernetes and easy settings management

Helm is a package manager for Kubernetes.  A package is called a chart and there are a lot of charts available. Your own application can be deployed to Kubernetes with a chart also. The advantage of using a chart regarding settings, is that the settings are external of this chart in a seperate settingsfile (called Values.yaml). Besides this, you can manage all settings of all Kubernetes artifacts that you deploy in this single Values.yaml file. This makes it easy to use in your CI/CD pipeline.

There is a guide to create a chart. This is a chart for the settingsdemo .NET Core 2 application. Most of it is generated by running

helm create mySettingsDemo

I’ve splitted Environment and Build specific variables in a seperate Values.dev.yaml file. This way the Values.yaml that belongs to the chart itself doesn’t have to change for every environment.
Deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "SettingsDemoChart.fullname" . }}
  labels:
    app: {{ template "SettingsDemoChart.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "SettingsDemoChart.name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: {{ .Values.service.internalPort }}
		  env:
		  - name: ServiceBus__Url
		   	value: {{ .Values.settingsDemo.serviceBus__Url | quote }}  
		  - name: ServiceBus__Key
			value: {{ .Values.settingsDemo.serviceBus__Key | quote }}
		  - name: Database__ConnectionString
			value: {{ .Values.settingsDemo.database__ConnectionString | quote }}
		  - name: ExtraSettingNotInSettingsFile
			value: {{ .Values.settingsDemo.extraSettingNotInSettingsFile | quote }}

Service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "SettingsDemoChart.fullname" . }}
  labels:
    app: {{ template "SettingsDemoChart.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.externalPort }}
      targetPort: {{ .Values.service.internalPort }}
      protocol: TCP
      name: {{ .Values.service.name }}
  selector:
    app: {{ template "SettingsDemoChart.name" . }}
    release: {{ .Release.Name }}

Values.yaml:

replicaCount: 1
image:
  repository: myregistry.azurecr.io/settingsdemo
  pullPolicy: Alway
service:
  name: settingsDemoService
  type: LoadBalancer
  externalPort: 80
  internalPort: 80

Values.dev.yaml:
The values are hardcoded in this sample, you can replace the variables with placeholders to use it in your CI/CD pipeline.

image:
  tag: 12
settingsDemo:  
  serviceBus__Url: "myServiceBus.servicebus.windows.net"
  serviceBus__Key: "abcdefghijklmnopqrstuvwzyz"
  database__ConnectionString: "mySqlConnectionstring"
  extraSettingNotInSettingsFile: "extraValue"

This chart can be deployed with:

helm install mySettingsDemo -f Values.dev.yaml

This is actually a very simple basic chart. Helm provides all kind of functions and flow control, like if/else statements to “generate” Kubernetes templates with much more flexibility.

During development there are other ways to handle variables and settings:

5. Environment Variables and settings during development

So if you configure secret settings in your settings files, the secrets can be seen by everyone who has access to your Docker container. How to use secrets then during development? You can configure Environment Variables in Visual Studio in the Launchsettings.json file. This Lauchsettings.json should be ignored by git. Use the latest git ignore file from Github for this.
.

"profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development",
        "ServiceBus__Url": "myServiceBus.servicebus.windows.net",
        "ServiceBus__Key": "abcdefghijklmnopqrstuvwzyz",
        "Database__ConnectionString": "mySqlConnectionstring"
      }
    }

If you like, you can also use Environment Variables in Windows. If you are using Docker Compose to run your Docker container:

6. Docker Compose and Environment Variables during development

When you debug your .NET Core application itself, the solution above works great. If you have enabled Docker support and debug the docker-compose project, you should specify Environment Variables in Docker compose.
You can add the Environment Variables in docker-compose.override.yaml

version: '3'

services:
  settingsdemo:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ServiceBus__Url=myServiceBus.servicebus.windows.net
      - ServiceBus__Key=abcdefghijklmnopqrstuvwzyz
      - Database__ConnectionString=mySqlConnectionstring
    ports:
      - "80"

Advertentie