Fixing The subscription is not registered to use namespace ‘Microsoft.xxx’

When you automatically deploy to a brand new subscription, for example using VSTS, not all resource providers are registered.

For example when you try to deploy Azure EventHub you will get the following exception:

The subscription is not registered to use namespace ‘Microsoft.EventHub’

After adding resources by hand in the Azure Portal the corresponding resource providers are registered, which is the reason why you will not always see this exception.
Instead of adding resources by hand the first time, an alternative is to register the resource providers with powershell.

You can register the single resource provider that you need with the following line:

Register-AzureRmResourceProvider -ProviderNamespace “Microsoft.EventHub”

If you like, you can also register all resource providers at once.
This can be done with the following script:

Select-AzureRmSubscription -SubscriptionName “”

$providerNamespaces = @(Get-AzureRmResourceProvider -ListAvailable) | ? {$_.RegistrationState -eq “NotRegistered” } | select ProviderNamespace

$providerNamespaces | foreach {
write-host $_.ProviderNamespace
Register-AzureRmResourceProvider -ProviderNamespace $_.ProviderNamespace
}
write-host “Finished!”

(make sure you are using the latest Azure PowerShell version. Otherwise you will be asked to confirm for every registration (which you can prevent by adding -Force))

Enterprise Azure ARM Templates

Lately I’ve given many workshops to all kinds of customers of my employer Xpirit about the automatic deployment of Azure resources. Mainly with VSTS.

I noticed customers would like to have ready to use ARM Templates.

resource-groupOf course there are the valuable Azure Quickstart templates (or via the portal)  which I use a lot. Sometimes these templates offer a complete solution. Mainly I use these templates to have, as the name tells, a quick start, for creating ARM templates fast.

Another way to get the ARM Template you wish is to download it after creating a resource through the Azure portal. These generated ARM Templates are nice for a reference, but are too generic. To use these templates in your automatic deployment pipeline you will have some work to do.

Besides this, I create ARM Templates a lot lately. For demo’s and for the Dutch Azure Meetup. I would like to reuse the templates easier myself and offer these templates to the community.

For these reasons I decided to start collecting the ARM Templates I have used to create resources in Azure. You can find the ARM Templates here. The templates are meant to be used right away.

For now ARM templates can be used per Resource type. You can use two types of ARM templates for the same resource.

One standard version where the user of the template decides what the name of the created resource will be.

Most enterprises I’ve visited don’t want to think about the way resources are named in Azure, but wants a consistent way of naming. When it’s possible to pass a name to a resource, they will end up with all kinds of naming conventions. For this reason the ARM templates are offered with a namingconvention also. This version of the Enterprise ARM templates passes metadata to tags of the resources also.

Infrastructure as Code and VSTS

Written by Peter Groenewegen and Pascal Naber for the Xpirit Magazine

Your team is in the process of developing a new application feature, and the infrastructure has to be adapted. The first step is to change a file in your source control system that describes your infrastructure. When the changed definition file is saved in your source control system it triggers a new build and release. Your new infrastructure is deployed to your test environment, and the whole process to get the new infrastructure deployed took minutes while you only changed a definition file and you did not touch the infrastructure itself.

Does this sound like a dream? It is called Infrastructure as Code. In this article we will explain what Infrastructure as Code (IaC) is, the problems it solves and how to apply it with Visual Studio Team Services (VSTS).

Infrastructure in former times
We have radically changed the way our infrastructure is treated. Before the change to IaC it looked like this: Our Operations team was responsible for the infrastructure of the application. That team is very busy because of all their responsibilities, so we have to request changes to the infrastructure well ahead of time.

The infrastructure for the DTAP environment was partially created by hand and partly by using seven PowerShell scripts. The order in which the scripts are executed is important and there is only one IT-Pro with the required knowledge. Those PowerShell scripts are distributed over multiple people and are partly saved on local machines. The other part of the scripts is stored on a network share so every IT-pro can access it. In the course of time many different versions of the PowerShell scripts are created because it depends on the person who wants to execute it and the project it is executed for.

directories2

Figure 1: A typical network share

The configuration of the environment is also done by hand.

This process creates the following problems:

  • Changes take too long before being applied.
  • The creation of the environment takes a long time and is of high risk, not only because manual steps can be easily forgotten. The order of the PowerShell scripts is important, but only a single person knows about this order.
  • What’s more, the scripts are executed at a particular point in time and they are updated regularly. However, it is unclear whether the environment will be the same when created again.
  • Some scripts are on the work machine of the IT-Pro, sometimes because it’s the person’s expertise area, and sometimes because the scripts are not production code. In either case, nobody else has access to it.
  • Some scripts are shared, but many versions of the same script are created over time. It’s not clear what has changed, why it was changed and who changed it.
  • It’s also not clear what the latest version of the script is. (See figure 1)
  • The PowerShell scripts contained a lot of code. The code does not only contain the creation of resources, but also checks whether resources already exist and updates them, if required.
  • The whole process of deploying infrastructure is pretty much trial and error.

As you can see, the creation of infrastructure is an error-prone and risky operation that needs to change in order to deliver high quality, reproducible infrastructure.

Definition of Infrastructure as Code
Infrastructure as Code is the process of managing and provisioning computing infrastructure and its configuration through machine-processable definition files. It treats the infrastructure as a software system, applying software

Infrastructure as Code characteristics
Our infrastructure deployment example has the following infrastructure provisioning characteristics, which will be explained in the following paragraphs:

  • Declarative
  • Single source of truth
  • Increase repeatability and testability
  • Decrease provisioning time
  • Rely less on availability of persons to perform tasks
  • Use proven software development practices for deploying infrastructure
  • Idempotent provisioning and configuration

Declarative

imperative-vs-declarative

Figure 2: Schematic visualization of Imperative vs Declarative

A practice in Infrastructure as Code is to write your definitions in
Definition of Infrastructure as Code Infrastructure as Code is the process of managing and provisioning computing infrastructure and its configuration through machine-processable definition files. It treats the infrastructure as a software system, applying software a declarative way versus an imperative way. You define the state of the infrastructure you want to have and let the system do the work on getting there. In the Azure Cloud, the way to use declarative code definition files are ARM templates. Besides the native tooling you can use a third party tool like Terraform to deploy declarative files to Classic Azure and to AzureRM. PowerShell scripts use an imperative way. In PowerShell you specify how you want to reach your goals.

Single source of truth
The infrastructure declaration files are placed in a source control repository. This is the single source of truth. All team members can see and work on the files and start their own version of the infrastructure. They can test it, and then commit changes to source control. All changes are under version control and can be linked to work items. The source control repository gives insight into what is changed and by whom.
The link to the work item can tell you why it was changed. It’s also clear what the latest version of the file is. Team members can easily work together on the same file.

Increase repeatability and testability
When a change to source control is pushed, this initiates a build that can test the change and after that publish an artifact. That will trigger a release which deploys your infrastructure. Infrastructure as Code makes your process repeatable and testable. After deploying your infrastructure, you can run standard tests to see if the deployment is correct. Changes can be deployed and tested in a DTAP pipeline.
This makes your process of deploying infrastructure reliable, and when you redeploy, you will get the same environment time after time.

Decrease provisioning time
Everything is automated to create the infrastructure. This results in short provisioning times. In many cases a deployment to a cloud environment has a lead time of 5 to 10 minutes, compared to a deployment time of days, weeks or even months.
This is accomplished by skipping manual tasks and waiting time in combination with high-quality, proven templates. The automation creates an environment that should not be touched by hand. It handles your servers like cattle instead of pets*. In case of problems there is no need to logon to infrastructure to see what is going wrong and trying to find the problem and fix it. Just delete the environment and redeploy the infrastructure to get the original working version.

Rely less on availability of persons to perform tasks
In our team, everybody can change and deploy the infrastructure. This removes the dependency on a separate operations team. By having a shared responsibility, the whole team cares and is able to optimize the infrastructure for the application.

This will result in more efficient usage of the infrastructure deployed by the team. Operations is now spending more time on developing software than on configuring infrastructure by hand. Operations is moving more to DevOps.

Pets vs Cattle
Is a widely used metaphor for how IT operations should handle servers in the cloud.
Servers are like pets You name them and when they get sick, you nurse them back to health.
Servers are like cattle You number them and when they get sick, you get another one.

Use proven software development practices for deploying infrastructure
When applying Infrastructure as Code you can use proven software development practices for deploying infrastructure. Handing your infrastructure in the same way you handle your code, helps you to streamline the whole process. You can start and test your infrastructure on each change. Using Source control as a team is a must. The sources that it contains should always be in the state in which they can be executed. This results in the need for tests such as unit tests.

Idempotent provisioning and configuration
Creating an idempotent provisioning and configuration for provisioning will enable you to rerun your releases at any time. ARM Templates are idempotent. This means that every time they will be executed the result will be exactly the same. The configuration is set to what you have configured in your definitions. Because the definitions are declarative, you do not have to think about the steps on how to get there; the system will figure this out for you.

Creating an Infrastructure as Code pipeline with VSTS
There are many tools you can use to create an Infrastructure as Code pipeline. In this sample we will show you how to create a pipeline which deploys an ARM template with a Visual Studio Team Service (VSTS) build and release pipeline. The ARM Template will be placed in a Git repository in VSTS. When you change the template, a build is triggered, and the build will publish the ARM template as an artifact. Subsequently, the release will deploy or apply the template to an Azure Resource group.

pipeline

Figure 3: VSTS source control, build and release

Prerequisite
To start building Infrastructure as Code with VSTS you need a VSTS account. If you don’t have a VSTS account, you can create one at https://www.visualstudio.com. This is free for up to 5 users. Within the VSTS Account you create, you then create a new project with a Git repository. The next step is to get some infrastructure definition pushed to the repository.

ARM template
ARM templates are a declarative way of describing your infrastructure. ARM templates are json files that describe your infrastructure and can contain 4 sections: parameters, variables, resources and output. To get started with ARM templates you can read Resource Manager Template Walkthrough.

It is possible to create ARM templates yourself by choosing the project type Cloud ? Azure Resource Group in Visual Studio. The community has already created a lot of templates that you can reuse or take as a good starting point. The community ARM templates can be found on the Azure Quickstart Templates. ARM templates are supported on Azure and also on-premise with Microsoft Azure Stack.

In our example we want to deploy a Web App with a SQL Server database. The files for this configuration are called 201-web-appsql-database. Download the ARM template and parameter files and push them in your Git source control repository in your VSTS project.

VSTS Build
Now you are ready to create the build. Navigate to the build tab in VSTS and add a new build. Use your Git repository as the source. Make sure you have Continuous Integration turned on. This will start the build when code is pushed into the Git repository. As a minimum, the build has to publish your files to an artifact called drop. To do this, add a Copy Publish Artifact step to your build and configure it like this:

code

Figure 4: ARM template in Git

buildpipeline

Figure 5: Copy Publish Artifact configuration

VSTS Release
The next step is to use VSTS Release for deploying your infrastructure to Azure. To do so, you navigate to release and add a new Release. Rename the first environment to Development and add the task Azure Resource Group Deployment to the Development environment. This task can deploy your ARM template to an Azure Resource group. To configure your task, you need to add an ARM Service Endpoint to VSTS. You can read how to do this in the following blogpost: http://xpir.it/mg3-iac4. Now you can fill in the remaining information, i.e.  the name of the ARM template and the name of the parameters file (fig. 6):

releasepipeline1

Figure 6: Azure Resource Group deployment configuration

releasepipeline2

Figure 7: Clone an environment in Release

DTAP
At this point you only have a Development environment. Now you are going to add a Test, Acceptance and Production environment. The first step is to create the other environments in VSTS release manager. Add environments by clicking the Add environment button or bycloning the development environment.

Infrastructure as Code will help you to create a robust and reliable infrastructure in a minimum of time.

Each environment needs separate parameters, so you need to create a parameter json file per DTAP environment. Each environment gets its own azuredeploy.{environment}.parameters.json file, where {environment} stands for development, test, acceptance or production.

releasepipeline3

Figure 8: Configure each environment to a different parameters file

The deployment can be changed to meet your wishes. For example, deploy to a separate Resource group in Azure per DTAP environment. Now you have your first version of an Infrastructure as Code deployment pipeline. The pipeline can be extended in multiple ways. The build can be extended with tests to make sure the infrastructure is configured as it is supposed to be. The release can be extended by adding approvers, which makes sure that an environment will only be deployed after an approval of one or more persons.

Conclusion
Infrastructure as Code will help you to create a robust and reliable infrastructure in a minimum of time. Each time you deploy, the infrastructure will be exactly the same. You can easily change the resources you are using by changing code and not by changing infrastructure.

When you apply Infrastructure as Code, everything should be automated, which will save a lot of time, manual configuration and errors. All configurations are the same, and there are no more surprises when you release your application to production. All changes in the infrastructure are accessible in source control.

Source control gives great insight in why and what is changed and by whom. A DevOps team that applies Infrastructure as Code is self-contained in running its application. The team is responsible for all aspects of the environment they are using. All team members have the same power and responsibilities in keeping everything up and running, and everybody is able to quickly fix, test and deploy changes.

peterenpascal

This article was published in the Xpirit Magazine #3, get your soft or hard copy here.

VSTS Task to create a SAS Token

The Create SAS Token task creates a SAS Token which can be used to access a private Azure Storage Container. The task also gets the StorageUri. Both variables can be used in subsequent tasks, like the Azure Resource Group Deployment task. This is the first task of the Infrastructure as Code serie.

The Task can be found in the marketplace and added to your VSTS account. The code is open source and can be found on GitHub.

Prerequisites for the sample

In this sample I’m executing an ARM template which uses linked ARM Templates. These linked ARM Templates are stored in a private Azure Storage Container. I will be using the Azure Resource Group Deployment task to deploy the parent ARM Template.

The Azure Storage Container looks like this:

AzureStorageContainer

The StorageAccount.json ARM Template looks like this:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountName": {
      "type": "string"
    },
    "storageAccountType": {
      "type": "string",
      "defaultValue": "Standard_LRS",
      "allowedValues": [
        "Standard_LRS",
        "Standard_GRS",
        "Standard_ZRS",
        "Premium_LRS"
      ],
      "metadata": {
        "description": "Storage Account type"
      }
    }
  },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[parameters('storageAccountName')]",
      "apiVersion": "2016-01-01",
      "location": "[resourceGroup().location]",
      "sku": {
        "name": "[parameters('storageAccountType')]"
      },
      "kind": "Storage",
      "properties": {
      }
    }
  ]
}

The ARM template which links to the StorageAccount.json looks like this:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountName": {
      "type": "string"
    },
    "_artifactsLocation": {
      "type": "string",
      "metadata": {
        "description": "Change this value to your repo name if deploying from a fork"
      },
      "defaultValue": ""
    },
    "_artifactsLocationSasToken": {
      "type": "securestring",
      "metadata": {
        "description": "Auto-generated token to access _artifactsLocation",
        "artifactsLocationSasToken": ""
      },
      "defaultValue": ""
    }
  },
  "variables": {
  },
  "resources": [
    { 
      "apiVersion": "2015-01-01",
      "name": "storage",
      "type": "Microsoft.Resources/deployments",
      "properties": {
        "mode": "Incremental",
        "templateLink": {
          "uri": "[concat(parameters('_artifactsLocation'),'/StorageAccount.json',parameters('_artifactsLocationSasToken'))]",          
          "contentVersion": "1.0.0.0"
        },
        "parameters": {
          "storageAccountName": {
            "value": "[parameters('storageAccountName')]"
          }
        }
      }
    }
  ],
  "outputs": {
  }
}

In this sample the ARM Template above is stored in Git. A build is responsible to create an artifact of this ARM Template, so it can be used in the Release. The release will be explained in the next paragraph.

Steps to use and configure the task

  1. Install the task in your VSTS account by navigating to the marketplace and click install. Select the VSTS account where the task will be deployed to.
  2. Add the task to your release by clicking in your release on add a task and select the Utility category. Click the Add  button on the Create SAS Token task.
    SelectSasTokenTask
  3. Configure the task. When the task is added the configuration will look like this
    EmptySasTokenTask
    All yellow fields are required.
    – Select an AzureRM subscription. If you don’t know how to configure this. Read this blogpost. (I’m using an Azure Principal with “reader” rights only on the ResourceGroup which contains the StorageAccount)
    – Select the Storage Account where you want to create a SAS Token for
    – Enter the name of the Storage Container
    – A variable for the SAS Token is also required. By default a variable is configured with the name storageToken.
    The configuration of the Create SAS Token Task looks like this:
    SasTokenConfiguration
    Note that I’m using an ServicePrincipal with readonly access to the ResourceGroup which contains the StorageAccount with a private container.
  4. Configure subsequent tasks which need the SAS Token. In the following sample the Azure Resource Group Deployment task is used.
    – Configure the task and fill the Override Template Parameters field with:

    -_artifactsLocation $(storageUri) -_artifactsLocationSasToken (ConvertTo-SecureString ‘$(storageToken)’ -AsPlainText -Force)

    The output variables of the Create SAS Token task are used here.
    The configuration of the Azure Resource Group Deployment task looks like this:
    DeployAzureResourceGroupTask
    Note that I’m using a different Service Principal. This Service Principal is specific for this project (and environment).

  5. Run the release

 

Create an Azure Service Principal and a VSTS ARM Endpoint

25-8-2016: Update because the UI to create a Service in VSTS changed

When you want to access Azure from VSTS there are multiple possibilities. It’s for example possible in VSTS to configure an Azure Classic Endpoint and after that configure the endpoint with credentials or with a certificate. The ARM way is to add an Azure Resource Manager Endpoint. To configure this you will need the settings of an Azure Service Principal. This blogpost tells you how to create both the Service Principal in Azure and the ARM Endpoint in VSTS.

Azure Service Principal

You can create an Azure Service Principal on multiple ways. Here you can read how to add an Azure Service Principal through the Classic Azure Portal. It’s also possible to add an Azure Service Principal through PowerShell. This PowerShell script can be used to create a Principal that has access to the whole subscription. I prefer to create a Service Principal that has access to a single ResourceGroup only. Here you can find the slightly changed script which support ResourceGroup access. When you run the script, the output will look like the following picture. Later on you will need the last five parts to create an ARM Endpoint in VSTS.

output

VSTS ARM Endpoint

To access the window where you can create an Endpoint in VSTS click on the Team Settings “Gear” icon in the upper right of the screen. After that click on the Services tab. Note that an Endpoint in VSTS is project specific.
TeamSettings

Alternatively, when you are configuring a task in VSTS which uses an Endpoint, you can click the Manage link. Like in the following screendump of the Azure WebApp Configuration task:
Manage

In both cases you should see the following screen and click on New Service Endpoint. In the menu select Azure Resource Manager.
services

The following popup screen will be shown. Because we don’t want a Service Principal in Azure which has access to the whole subscription, don’t click OK. Instead click on the here link.

Create endpoint 1

The following screen appears. Here you can fill in all the information that the PowerShell script outputs.
Create endpoint 2

 

 

The Connection Name is the only field where you can enter custom information. The other fields must match exactly.
linked

There is no validation or connection checker on the information that you fill in. When you use the Azure WebApp Configuration task for example and have selected an ARM Endpoint. The task selects all webapps where the Service Principal has access to. When no WebApps appears (and there should), the entered information will probably be wrong.

VSTS Task to deploy AppSettings and ConnectionStrings to an Azure WebApp

The Azure WebApp Configuration task reads VSTS variables and adds those as AppSettings and ConnectionStrings to an Azure WebApp. The task also supports Slot Settings. The task can be linked to a web.config to validate if all AppSettings and ConnectionStrings in the web.config exists as VSTS variable.

The Task can be found in the marketplace and added to your VSTS account. The code is open source and can be found on GitHub.

Naming convention

The task uses naming conventions in the VSTS variables to deploy appsettings and connectionstrings to an Azure WebApp. If you like as a Slot Setting. The value of the VSTS variable is used for the value for the appsetting or in case of a connectionstring, the connectionstring.

The following naming conventions rules are supported:

  • The name of a variable for an appsetting must start with appsetting.
  • The name of a variable for a connectionstring must start with connectionstring.
  • The type of database should be added in the namingconvention as stated in the following table.
  • For a slotsetting the convention .sticky must be used.

The table below shows some examples:

Type Example VSTS variable name
appsetting appsetting.mysetting
appsetting with slotsetting appsetting.mysetting.sticky
connectionstring to SQL Server connectionstring.myconnectionstring.sqlserver
connectionstring to SQL Azure connectionstring.myconnectionstring.sqlazure
connectionstring to custom connectionstring.myconnectionstring.custom
connectionstring to MySQL connectionstring.myconnectionstring.mysql
connectionstring with slotsetting connectionstring.myconnectionstring.sqlserver.sticky

Steps to use and configure the task

  1. Install the task in your VSTS account by navigating to the marketplace and click install. Select the VSTS account where the task will be deployed to.
  2. Add the task to your release by clicking in your release on add a task and select the Utility category. Click the Add  button on the Apply variables to Azure webapp task.
    addtask
  3. Configure the task. When the task is added the configuration will look like this:
    cleantask
    All yellow fields are required.
    – Select an AzureRM subscription. If you don’t know how to configure this. Read this blogpost.
    – Select the web app name.
    – Select the resourcegroup.
    – If you want to deploy to a Deployment slot, check the Deploy to Slot checkbox and select the Slot.
    – If you want to validate the VSTS variables against the appSettings and ConnectionStrings in the web.config of the application you deploy, then select the web.config. Otherwise uncheck the Validate variables checkbox.
    – When you want validation, you can choose how the task should behave when it finds incorrect variables. Default behavior is that the task will fail with an error. This results in a failed Release  and the variables will not be deployed.
    The other validation result action is  to only get a warning. The variables will be deployed to Azure.
    – By default all existing AppSettings and ConnectionStrings in Azure are overwritten by the variables in the release. When you don’t want this, but you want to preserve your appsettings or connectionstrings in your WebApp, then uncheck the Overwrite existing configuration checkbox.
  4. The web.config of your application is not being used to deploy variables to Azure. If you have configured validation, the keys of the appSettings and the names of the ConnectionStrings are used to validate if there are VSTS variables available for these settings. The web.config in this sample looks like this:
    webconfig
  5. Add variables that match the names of the variables in the AppSettings and ConnectionStrings. Also apply the namingconventions for the VSTS variables.
    vstsvariables
  6. Instead of configuring Release variables as in step 5, you can also configure Environment variables. This step shows you how you can configure the release in VSTS to support multiple environments. In this sample each environment will have the same appsettings and connectionstrings. But the value of those differentiate.
    After creating the Environments, configure variables per environment. Just like step 5, just add the keys and the values. ConfigureVariablesPerEnvironment
    To see the Environment variables at a glance, select the Environment variable view as you can see in the following picture:
    SelectEnvironmentVariables
    When you have configured all Environments, you will see an overview like the following:
    VariablesPerEnvironment
  7. When you run the release, the settings will be deployed to Azure. In this sample it looks like this:
    Note that the hidden value in VSTS is visible now in Azure.
    azure

Started Dutch Azure Meetup

Logo Meetup
This week Marco Mansi, Sander Molenkamp and I started the Dutch Azure Meetup: http://www.meetup.com/Dutch-Azure-Meetup/

Azure offers a lot of services and features, and this grows almost daily. It’s hard to be up-to-date on all the Azure features. This meetup gives you the opportunity to learn about Azure in a practical way.

We want to offer a platform for Azure enthusiasts, where you can get hands-on with the technology, talk with and learn from each other. The meetup will not handle customer cases in theory but instead we talk about deep technical details, real world experiences and best practices.

Our first meetup is planned for the 2nd of June and is all about Azure Container Service:

Join the first Dutch Azure Meetup!

Docker is taking the world by storm and Microsoft Azure Container Service is where Azure and Docker meet!

During this first meetup Mark van Holsteijn will give you a brief introduction into Docker and Microsoft Container Service.After which you will have ample opportunity to get hands-on with Docker, Marathon, Azure and ASP.NET core.

We will show you how Azure Container Service provides an excellent platform for creating cross-cloud applications.

About the speaker

Mark van Holsteijn is Principal Consultant at Xebia specialized in cloud software architectures with open-source technologies.

Agenda

18:00: Dinner

18:45: Word of welcome Dutch Azure Meetup

18:50: Azure Container Service (by Mark van Holsteijn)

19:30: Hands-on with Docker and Azure Container Service