Persistent Storage and Volumes using Kubernetes on Azure with AKS or Azure Container Service

Many applications hosted in a Docker container need a volume to store data on or to read from. The data can’t be stored in the Docker container itself because the data will be lost after a restart or when the container crashes. Persistent Storage has an independent lifecycle of a Pod. This blogposts shows the most used possibilities to use persistent storage using Kubernetes on Azure.

Let’s first see how to use a volume using Docker on your Windows PC.

Docker Volume

We begin without Kubernetes, let’s first see what we want to accomplish on our own local machine.

  1. Here on my github account you can find a simple .NET Core project. This Console application writes the date and time every 5 seconds to a file and to the Console. This file should not be available in a Docker Container, because when we stop the Docker container, we will loose our file. This is because Docker Containers store data ephemeral. The file should be written to the filesystem of the host. In this case, Windows. Clone this Github project.
  2.  Publish the project to directory obj/Docker/publish
  3. In the command prompt, in the directory of the project, run
    docker build . -t pnimages.azurecr.io/storagedemo
  4. Create directory c:\storage
  5. Now run the Docker Container. Mount a volume with the name storage to the Docker container.
    docker run -d -it -v c:/storage:/storage pnimages.azurecr.io/storagedemo. Every 5 seconds the date and time will be written to c:\storage\log.txt. When we stop the container, the file stays on it’s location. So we have persistent storage. This is what we want using Kubernetes.

Prerequisite for Kubernetes

To use the Storagedemo Docker image in Kubernetes, the Docker Image needs to be available in a Docker registry. I’ve pushed the image to my Azure Container Registry (ACR). My Container Registry it’s name is pnimages. Replace this name with the name of your ACR.

  1. Add an extra tag to the image with a version number:
    docker tag pnimages/storagedemo pnimages.azurecr.io/storagedemo:1
  2. login to Azure Container Registry:
    docker login pnimages.azurecr.io -u pnimages -p [password]
  3. docker push pnimages.azurecr.io/storagedemo
  4. docker push pnimages.azurecr.io/storagedemo:1
  5. Create a secret in Kubernetes with the credentials of the ACR:
    kubectl create secret docker-registry pnimages –docker-server=https://pnimages.azurecr.io –docker-username=pnimages –docker-password=[password] –docker-email=pnaber@xpirit.com

Kubernetes Volume

For Azure there are 2 kinds of Volumes available in Kubernetes. AzureDisk and Azurefile. AzureDisk maps to a vhd. AzureFile maps to a directory in an Azure Storage Account on a Fileshare.

Kubernetes Volume with Azure disk

What we will create looks like this:
Kubernetes Volume

The steps to use AzureDisk:

  1. Create an Azure Storage Account in any resourcegroup you like, but in the same location as your Kubernetes cluster.
  2. Create a File Share with the name storage
    KubernetesVolume-StorageAccount
  3. Get the name of the storage account and the key of the storageaccount. And for both the account name and the key get a base64 encoded value. I’ve used https://www.base64encode.org/ to do this.
  4. Create a secret in Kubernetes. Replace the value of azurestorageaccountname and azurestorageaccountkey with the values you obtained in step 3.
    apiVersion: v1
    kind: Secret
    metadata:
      name: volume-azurefile-storage-secret
    type: Opaque
    data:
      azurestorageaccountname: [base64encodedStorageAccountName]
      azurestorageaccountkey: [base64encodedStorageAccountKey]
    

    kubectl -f apply secret.yaml

  5. Deploy the Docker image with a Deployment of a pod:
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: volume-azurefile-storage-deployment
    spec:  
      template:
        metadata:
          labels:
            app: storagedemo
        spec:
          containers:
          name: storagedemo
            image: pnimages.azurecr.io/storagedemo:1  
            volumeMounts:
            name: azurefileshare
              mountPath: /storage      
          imagePullSecrets:
          name: pnimages       
          volumes:
          name: azurefileshare
            azureFile:
              secretName: volume-azurefile-storage-secret
              shareName: storage
              readOnly: false
    

    kubectl -f apply deployment.yaml

  6. The Docker container will start immediately and when you take a look in the fileshare of the Azure Storage account, a log.txt file will written.

Advantages and disadvantages:

+ you can use a storage account in any resourcegroup you like (must be in the same region though)
– have to do some work to create the secret with the base64 encoded values
– A Pod is directly coupled with a Volume, so the Pod has knowledge about the underlying cloud.

Kubernetes Persistent Storage

Kubernetes has a concept called StorageClass. With StorageClasses administrators can offer Profiles regarding storage. For example a profile to store data on a HDD named slow-storage or a profile to store data on a SSD named fast-storage. The kind of storage is determined by the Provisioner. For Azure there are 2 kind of provisioners: AzureFile and AzureDisk.

The big difference between AzureFile and AzureDisk is the AccessMode. There are 3 AccessModes.
ReadWriteOnce – the volume can be mounted as read-write by a single node
ReadOnlyMany – the volume can be mounted read-only by many nodes
ReadWriteMany – the volume can be mounted as read-write by many nodes

AzureFile supports all three. AzureDisk supports ReadWriteOnce only.
In the situation where a pod is configured to use a volume to write or read data to and is being restarted on a different host for whatever reason, you can’t use AzureDisk.

With the use of AzureDisk a vhd is attached to the node. The size of the node is dependent on the number of attached vhd’s. As a general rule you can have twice as many attached disks as CPU cores.

AzureFile uses SMB drives. Regarding performance is this not suitable to store the data of databases for example.

If you have multiple Pods of the same image (multiple replicas) that form a group that work together and you want to mount a volume per Pod to write data to. You may want to use StateFull Set. This is a subject for another blogpost.

Azure Container Service with Kubernetes offers StorageClasses out of the box:
kubectl get storageclass
DefaultStorageClasses
AKS has a different number of out of the box StorageClasses.

As you can see all 3 StorageClasses in ACS are azure-disk type of storage. There is a standard storage to store to HDD and a premium to store to SSD. There is also 1 default storage. This one stores to HDD also and is marked as default StorageClass. It is possible to mark another StorageClass as default.

There is no name of a Storage Account configured in the StorageClasses above. When one of the StorageClasses is used, a Storage Account with a generated name is provisioned and used.

To use a specific storageAccount you can create a StorageClass yourself. This StorageClass can optionally reference a specific StorageAccount and uses the azure-file provisioner to use fileshares.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefilestorage
provisioner: kubernetes.io/azure-file
parameters:
  storageAccount: pnstandarddatastorage

Kubernetes Dynamic Persistent Volume (with non-default StorageClass)

We are only going to provision a PersistentVolumeClaim and a Pod. The PersistentVolume and Secret is created Dynamic. Because we are not using the default StorageClass, we need to create a custom StorageClass also. This StorageClass can be reused for multiple deployments.
KubernetesDynamicPersistenceVolume

  1. Make sure you have created the StorageAccount. This StorageAccount must be located in the same ResourceGroup as the Cluster!
  2. Create the StorageClass and reference the name of the existing StorageAccount
  3. A PersistentVolumeClaim (PVC) can request a size and specify an accessmode.
    Deploy the PersistentVolumeClaim. The created StorageClass is referenced.

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: dynamic-persistence-volume-claim
    spec:
      accessModes:
        - ReadWriteMany
      storageClassName: azurefilestorage
      resources:
        requests:
          storage: 5Gi
    

    When we deploy a PersistentVolumeClaim without the line storageClassName: azurefilestorage the default StorageClass is used! A StorageAccount is provisioned in the same ResourceGroup as the cluster.

    After creating this PersistentVolumeClaim (PVC), Kubernetes creates a new Secret with access parameters to the StorageAccount. The same kind of Secret we created earlier ourselves with the base64 encoded values. The name is azure-storage-account–secret

    Not only the PVC is created, also a PersistentVolume is created by Kubernetes. The name is pvc-.
    A FileShare is created also. Located in the StorageAccount with a generated name.

  4. Now we are ready to deploy our Docker container:
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: dynamic-persistence-volume-deployment
    spec:  
      template:
        metadata:
          labels:
            app: storagedemo
        spec:
          containers:
          name: storagedemo
            image: pnimages.azurecr.io/storagedemo:1  
            volumeMounts:
            name: azurefileshare
              mountPath: /storage      
          imagePullSecrets:
          name: pnimages       
          volumes:
          name: azurefileshare
            persistentVolumeClaim:
              claimName: dynamic-persistence-volume-claim
    

If you want to use the Default StorageClass, just leave the line: storageClassName: azurefilestorage from the PersistentVolumeClaim

Advantages and disadvantages:

+ The secret is generated. No manual base64 encoding needed.
+ You decide the name of the StorageAccount
+ Create StorageClass for reuse and every deployment you create a Pod and a PersistentVolumeClaim only
+ Deployment of the Pod is Cloud agnostic, the Pod doesn’t know anything about the underlying cloud.
– The Azure Storage Account has to be in the same ResourceGroup as the cluster
– The name of the FileShare in the Storage Account can’t be configured. It’s name is generated.
– No reuse of PersistentVolume. Every Deployment results besides a PersistentVolumeClaim, also in a PersistentVolume. This is because the PersistentVolume is created with the requested amount specified in the Claim. So the StorageVolume is never created with a larger size. This is the reason why there is always created a new PersistentVolume for the new Claim. (It’s technically possible to reference the same PersistentVolumeClaim from the Pod)

Static Persistent Volume

When static Persistence is used, a PersistentVolume is created manually. This way the administrator of the cluster can make Storage available to developers of Pods on a managed way.
KubernetesStaticPersistenceVolume

    1. Make sure you have created the StorageAccount. This StorageAccount must be located in the same ResourceGroup as the Cluster!
    2. Create the StorageClass and reference the name of the existing StorageAccount. This is commonly done by the cluster administrator.
    3. Create a PersistentVolume. This is commonly done by the cluster administrator.
      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: static-persistence-volume
        labels: 
          storage: slow
      spec:
        capacity:
          storage: 50Gi
        accessModes:
        - ReadWriteOnce
        storageClassName: azurefilestorage
        azureFile:
          secretName: static-persistence-secret
          shareName: storage
          readOnly: false
      
    4. Create the secret that is referenced in the PersistentVolume
      Both the azurestorageaccountname and azurestorageaccountkey should be base64 encoded.

      apiVersion: v1
      kind: Secret
      metadata:
        name: static-persistence-secret
      type: Opaque
      data:
        azurestorageaccountname: [base64encodedStorageAccountName]
        azurestorageaccountkey: [base64encodedStorageAccountKey]
      
    5. Now the steps look similar to the dynamic way.
      Create a PersistentVolumeClaim. Note that the storageClassName matches the storageClassName in the PersistentVolume.

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: static-persistence-volume-claim 
      spec:
        accessModes:
          - ReadWriteMany  
        resources:
          requests:
            storage: 5Gi
        storageClassName: azurefilestorage
      
    6. Create the Pod:
      apiVersion: extensions/v1beta1
      kind: Deployment
      metadata:
        name: static-persistence-volume-deployment
      spec:  
        template:
          metadata:
            labels:
              app: storagedemo
          spec:
            containers:
            name: storagedemo
              image: pnimages.azurecr.io/storagedemo:1  
              volumeMounts:
              name: azurefileshare
                mountPath: /storage      
            imagePullSecrets:
            name: pnimages       
            volumes:
            name: azurefileshare
              persistentVolumeClaim:
                claimName: static-persistence-volume-claim 
      

Advantages and disadvantages:

+ You have all naming in control. For the FileShare, the StorageAccount
+ Possible to apply seperation of concerns regarding roles. Cluster administrator vs Developers
+ Deployment of the Pod is Cloud agnostic, the Pod doesn’t know anything about the underlying cloud.
– The Azure Storage Account has to be in the same ResourceGroup as the cluster
– Relatively a lot of work to configure everyting including the Secret.
– If you make a mistake in naming and no StorageVolume can be matched by the StorageVolumeClaim, a new StorageVolume is generated.

Alle files can be found on this github repo.

Advertenties

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen.

WordPress.com logo

Je reageert onder je WordPress.com account. Log uit /  Bijwerken )

Google+ photo

Je reageert onder je Google+ account. Log uit /  Bijwerken )

Twitter-afbeelding

Je reageert onder je Twitter account. Log uit /  Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit /  Bijwerken )

w

Verbinden met %s