Migrating Terraform from AWS to Azure: credentials & secrets

It is probably safe to say that the vast majority of Terraform users cut their teeth with the AWS providers to deploy their first cloud environments using infrastructure as code. It was the way that I started to use IaC against cloud infrastructure. Part of this is the ease of access to the AWS ecosystem. The ability to use the environments free tier indefinitely. But there comes a time that you realize there are other clouds out there. This post looks at migrating Terraform from AWS to Azure: credentials & secrets.

Multi-clouding your Terraform code

How do you change your code to deploy your infrastructure to another cloud provider?

Today we are going to look at moving the environment to Azure and GCP. It is true that Terraform is touted as one code to rule all deployments but although this concept is correct at a high level, it is not as simple as just changing the Terraform provider from the AWS one to the Azure one.

This leads us down another contentious path, is Terraform really cloud-agnostic if all the Terraform files and modules have to be re-written or modified to work? For example, in the original lamp script, we used the “AWS” provider and all the resources are AWS specific like “aws_subnet” and “aws_instance” and so on.

In fact, even to build out this relatively simple flat file terraform deploy requires an in-depth knowledge of AWS and its workings. The same is the case when building against another different cloud provider. We cannot simply change the resource name “azure_subnet” or “azure_instance”. We need to have a good understanding of what is needed to create these objects in Azure. So with that in mind let us start our transformation.

The first thing we need to do is to use Azure Cloud Shell to allow Terraform access to do so login to your Azure environment.

If you have never accessed Cloud shell before you will see split happen in your console at the bottom of the screen and a statement welcoming you to  you Azure Cloud shell

You now have a choice of Bash or Powershell, as we are attempting to be as agnostic as possible click the Bash link.

Under subscription chose the correct environment and click the “Create Storage” button.

To login directly to Azure CLI, you need to enter the command az login.

As we are already logged in to Azure we will receive the following response. Take a not of the code from your command issuance as you will need it. For security reasons, we have greyed ours out.

On successful login you will receive the following response in your web browser:

On your console, you will see something similar to the following.

However to login into Azure with Terraform you will need to create a Service Principal account.

An Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.

Creating a Service Principal Account

There are many options when creating a Service Principal account for Terraform use we need to create ours with the Contributor permission level, this will give us full permissions to read and write to the Azure account.

az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/<subscription_id>"

remember to replace the subscription_id with the bonafide subscription_id.

Now that we have created our account lets head out of the Azure Console and install Azure CLI on our local machine. Once again we are back on a Windows 10 machine the installation package is a simple MSI file downloaded from here.

Once you have successfully installed Azure CLI you should open a CMD prompt and login to Azure as your Service Principal account using the details shown in image above for “username -u” “password -p” and “–tenant”

Take a note of the details shown and issue an az account show command the details should match

Now that we have successfully logged into our Azure environment using the CLI it is time to configure the environment to use terraform.

Configuring Terraform to use Azure

Now be careful, because if you blink you will miss this.

Create a directory on Clouddrive if you are going to do everything thought the Azure portal or a web-browser.

Configuring your remote terraform environment

If you are configuring to allow your personal device to deploy Terraform into Azure then there are a few more hoops to jump through.

You still need to configure your provider however there are four more entities you need to provide to allow authentication to your cloud.

There are:

• subscription_id
• client_id
• client_secret
• tenant_id

so now our test.tf file looks similar to the below:

provider "azurerm" {
# The "feature" block is required for AzureRM provider 2.x.
# If you are using version 1.x, the "features" block is not allowed.
version = "~>2.0"

    subscription_id = "00000000-0000-0000-0000-000000000000"
    client_id = "00000000-0000-0000-0000-000000000000"
    client_secret = "secret here "
    tenant_id = "00000000-0000-0000-0000-000000000000"

features {}
}

resource "azurerm_resource_group" "rg" {
    name = "myRemoteAmazicTest-rg"
    location = "northeurope"
}

If you remember from your AWS provider, you will remember that there are only two entities you need to provide for authentication; that of your access_key and your secret_key.

Now that you have configured the environment, it is time to test the connectivity. Issue your “terraform init” and your “terraform apply”. Then look in your Azure console and you will see a resource group called “myRemoteAmazicTest-rg”. Finally, issue a “terraform destroy” to remove it.

Now that we have connected to Azure with Terraform, and proved that it works, let’s, as promised earlier in this post, flip back and reconfigure the authentication mechanism to use HashiCorp Vault to provide one-time use tokens and do security properly.

Configuring for Vault

Logging into Azure as a user when using Vault will obviously change the authentication flow. This was also the case when we implemented Vault to provide one-time tokens for AWS Terraform deployments. Rather than using a direct connection to Azure AD and the Service Principal accounts now, we will be using Vault to assume the role of the user.

There is an assumption is that there is a working HashiCorp Vault server in your environment, if not revisit this post on how to configure one.

Each instance will have unique credentials that are not shared with anybody, further those credentials are short-lived so you have reduced that potential for your credentials being compromised. And if they are compromised the potential for misuse has been reduced.

As a refresher, Vault is policy-driven, and any access policy that is empty has by default no permissions, this means that Vault as a whitelist service rather than a blacklist service. Basically this means, that everything is denied unless explicitly granted. So it is secure by default.

For the purpose of this guide, we will be using the pre-configured service Principal account created earlier in this article. Collect your Subscription_ID, Tenant_ID, Application_Id, and we will be setting up a new client secret just for Vault. You should now click “Certificate & Secrets”

Next select “New Client Secret” you will find this under the section “Client Secrets”, you will receive the following form, Insert a description, make it understandable so that those that come after will understand what it is.

Select the validity of the secret. This depends on the policies in your company. Click add and copy and store the generated secret value which is your client secret. This is important as you will only see this once.

Next, we need to configure API permissions, so from the options on the left of the screen click API Permissions

Click “Add Permission” and then select “Azure Active Directory Graph” this can be found under “Supported Legacy APIs”. Next click Delegated permissions, expand User, and then select the check-box for User.Read

Next, we need to configure the Applications Permissions, click on the Box titled Application Permissions

Click Application permissions, expand Application, and Directory.

Select the check-box for Application.ReadWrite.All and Directory.ReadWrite.All.

Finally, click API permissions button at the bottom to save the changes

Finally, navigate to your Subscriptions service blade. Now, click into your Subscription name and click Access control (IAM), finally click Add under the Add a role Assignment. And configure it as shown, replacing the username for the one in your environment.

Click Save to confirm the changes. Unless you neglected to previously configure the Service Principal account, this save will fail as we have previously added the contributor role to the terraform SP.

Next we move onto configuring Vault to speak to Azure

Enabling the Azure Secret Engine

Login to your vault server and select the Secrets Tab, click the “Enable new Engine”.

Select the Azure radio button and click next.

Finally click “Enable Engine” to activate.

this will be mounted to the default secret engine path of “azure/”

we could have done all this from the vault CLI with the command

vault secrets enable azure

if we wished to give a custom Azure path we would have used the -path argument.

Next, we need to configure the secret engine with account details this is where those details we collected earlier come into use

It is quicker to use the command line for this procedure. Just for clarity, this command is inputted on a single line.

vault write azure/config \
subscription_id=$AZURE_SUBSCRIPTION_ID \
tenant_id=$AZURE_TENANT_ID \
client_id=$AZURE_CLIENT_ID \
client_secret=$AZURE_CLIENT_SECRET

Configuring AppRoles for Terraform

AppRoles are the method that allows machines of applications to authenticate with Vault defined roles, it represents a set of policies and login constraints that must be net to allows a token receipt. This is a rather complicated and multi-faceted process and to be fair the HashiCorp documentation is not particularly clear.

The first thing we need to create our role for Azure, again this will be configured at the command-line. We start by enabling the approle authentication method

Vault auth enable approle

Next we need to create named role in our case we will create a role called “Azure-Terraform”.

One thing to note here is that at the time of writing the Vault documentation of the site is incorrect, the documentation here has a massive gap, and in the more in-depth documentation here they inform you to use the token_num_uses=10, this is actually incorrect as it creates tokens that are restricted and cannot create the necessary child token needed. The correct usage is “token_num_uses=0”

So let’s create a role called Azure-Terraform

vault write azure/roles/Azure-Terraform application_object_id=<Your_app_obj_id_here> ttl=1h token_num_uses=0

If this fails the first time then the application_object_id is incorrect thanks to Kalafut on GitHub for this fix

From a machine with Azure CLI installed and configured run the following commands

az login --service-principal --tenant $AZURE_TENANT_ID -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET
az ad app show --id $APP_ID | jq .objectId

and replace the UUID with the value received from the second command and issue the vault command again.

NOTE jq is a JSON parser and is available from https://stedolan.github.io/jq/

Verify that all is good with your newly created role by issuing the following command

vault read azure/creds/<your role name here>

There you go a credential and a secret, that will have expired by the time you read this. To confirm this is working correctly reissue the command to verify that a different secret_id has been created.

Here are those Leases in Vault Console

You have your credentials but can’t do anything with them yet as we need to tell Terraform to use them.

Creating an AppRole in Vault

We don’t need no policy! Oh yes you do.

Before we configure our AppRole we need to assign a policy to it. This policy effectively maps our Azure-Terraform credentials with the policy to create a child token. For this, we have to give read permissions to the correct account and then have it update the create the token. You can create this from the command line but it is actually easier to do it on the web-console

path "azure/creds/Azure-Terraform" {
capabilities = ["read"]
}
path "auth/token/create" {
capabilities = ["update"]
}

Click on the Policies tab

Next name your policy and place the code above in the edit box and press Create policy to complete.

Return to the Access tab, select Entities and click Create Entity

Enter your AppRole name and assign the policy you created earlier and click create.

Verify everything is correct by clicking on policies

you should see something similar to what is shown is shown above.

Configuring Terraform to use Vault for Azure Authentication

OK you have now hopefully fought your way through the

If you cast your mind back to when we introduced vault into the authentication path for the AWS deployments we need to configure the Vault provider. The code below should be recognizable as it is very similar to the code that was used in previously for AWS authentication, with a number of extra bits and pieces for Azure authentication.

provider “vault” {
    address = var.vault_addr
auth_login {
    path = “azure/creds/Azure-Terraform”
        parameters = {
            role_id   = var.login_approle_role_id
            secret_id = var.login_approle_secret_id
            }
        }
}

Obviously replace the “path” to match your credentials.
Now there are a number of new things here. The path statement which points to the role you created earlier, and the parameters role_id and secret_id.

Finally, we need to add the following code

data “vault_generic_secret” “azure” {  
    path = “azure/creds/Azure-Terraform”
}

Now you would think that was that but now Azure has a particular foible in that causes the login to fail. This is because Azure AD like local AD is a distributed service and there is no guarantee that your token login request will be presented to the exact same node that created it, but it will land at a node that the credentials have not been replicated too. So we need to create an artificial delay in the login process.

To do this we are going to introduce another special resource, this is the external resource.

data "external" "subscription_id" {
    program = ["./install.sh", "5f03aebb-6cf7-42c1-ad90-1d13a2f73174", "512"]
}

This particular code block allows Terraform to import an external data point. In our case we are running a script to introduce a 512-millisecond delay into the authentication process.

Your script look will like this if you are using Linux or MacOS

#!/usr/bin/env bash
subscription_id=$1
sleep $2
echo “{ \”subscription_id\”: \”$subscription_id\” }”

So now you main.tf file should look something like this:

provider "azurerm" {
   version = "~>2.0"
   subscription_id =   data.external.subscription_id.result["subscription_id"]
   tenant_id = "<Tenant_ID>"
   client_id = data.vault_generic_secret.azure.data["client_id"]
   client_secret = data.vault_generic_secret.azure.data["client_secret"]
 
   features {}
}

provider "vault" {
    address = "https://ec2-18-208-156-219.compute-1.amazonaws.com:8200/"
    auth_login {
    path = "auth/approle/login"
        parameters = {
            role_id = "<Role_ID>"
            secret_id = "<Secret_ID>"
        }
    }
}

data "external" "subscription_id" {
    program = ["./install.sh", "<subscription_ID>", "512"]
}

data "vault_generic_secret" "azure" {
   path = "azure/creds/Azure-Terraform"
   }

resource "azurerm_resource_group" "rg" {
    name = "myRemoteAmazicTest-rg"
    location = "northeurope"
}

When we run a plan against you will receive something similar to the below.

If you receive this error:

AADSTS7000215 you will need to increase your timeout period. In the data “external” code block. I had to increase mine to 512 before I started to get consistent logon results.

Summary

We covered migrating Terraform from AWS to Azure: credentials & secrets are arguable the most tedious part to migrate as AWS and Azure are so wildly different. Now that we have set up our environment for secure Terraform deploys against Azure, the next post in the Terraform on Azure series will start reworking the code of the LAMP stack deployment and deploy the environment on Azure.