Prod1gy's Arena
Azure DevOps Pipeline with Terraform

Azure DevOps Pipeline with Terraform

Prod1gy's photo
Prod1gy
·

8 min read

Table of contents

Architecture Design

What is Azure DevOps?

Azure DevOps is a Software as a service (SaaS) platform from Microsoft that provides an end-to-end DevOps toolchain for developing and deploying software. It also integrates with most leading tools on the market and is a great option for orchestrating a DevOps toolchain. At DevOpsGroup, we have lots of customers who have found Azure DevOps fits their needs irrespective of their language, platform or cloud.

Azure DevOps comprises a range of services covering the full development life cycle. At the time of writing these are:

  • Azure Boards: agile planning, work item tracking, visualization and reporting tool.

  • Azure Pipelines: a language, platform and cloud-agnostic CI/CD platform with support for containers or Kubernetes.

  • Azure Repos provides cloud-hosted private git repos.

  • Azure Artifacts: provides integrated package management with support for Maven, npm, Python and NuGet package feeds from public or private sources.

  • Azure Test Plans: provides an integrated planned and exploratory testing solution.

Azure DevOps can also be used to orchestrate third-party tools.

What is Terraform?

Terraform is an infrastructure as a code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently.

HashiCorp Terraform is an infrastructure as a code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version reuse, and share. You can then use a consistent workflow to provision and manage all of your infrastructure throughout its lifecycle. Terraform can manage low-level components like computing, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.

How does Terraform work?

Terraform creates and manages resources on cloud platforms and other services through its application programming interfaces (APIs). Providers enable Terraform to work with virtually any platform or service with an accessible API.

The core Terraform workflow consists of three stages:

  • Write: You define resources, which may be across multiple cloud providers and services. For example, you might create a configuration to deploy an application on virtual machines in a Virtual Private Cloud (VPC) network with security groups and a load balancer.

  • Plan: Terraform creates an execution plan describing the infrastructure it will create, update, or destroy based on the existing infrastructure and your configuration.

  • Apply: On approval, Terraform performs the proposed operations in the correct order, respecting any resource dependencies. For example, if you update the properties of a VPC and change the number of virtual machines in that VPC, Terraform will recreate the VPC before scaling the virtual machines.

High-level overview:

  1. Provision Azure Backend

  2. Create the Terraform Template

  3. Prepare the Azure Devops Organisation

  4. Create CI Pipeline

Steps:

  1. Create an Azure DevOps organization

  2. Select New organization.

  3. Confirm the information, and then select Continue.

  4. Sign in to your organization (https://dev.azure.com/{yourorganization}).

  5. Select New project.

  6. Select Create. Azure DevOps displays the project welcome page.

  7. Select Organization settings and then select Projects.

  8. My Azure DevOps repo contains a starter YAML pipeline.

    and I created a subfolder called MainDir.

    YAML pipelines provide a more efficient and scalable way to manage your build and release pipelines in Azure DevOps.

    In a YAML pipeline, you define the tasks, steps, and stages of your pipeline in a YAML file, and then check it into your source control repository. The MainDir folder is to create your terraform script that the pipeline will use to execute.

  9. For your pipeline to create resources into your azure subscription, you need to create an Azure service principal that has the permissions to create appropriate resources such as resource group, storage account and azure function app.

    An Azure AD service principal is a security identity that represents an application or service in Azure and enables it to interact with Azure resources in a secure and controlled manner.

    It is used to authenticate and authorize applications and services that need to access Azure resources and to manage their permissions and access to those resources.

    I created a service principal to be used by Azure DevOps Service Connection using Azure Cloud Shell.

    Creating SP using azure cloud shell

    export servicePrincipalName=AzTera export roleName="Contributor" export subscriptionID=xxxxxxxx-xxxx-xxxx-b108-8299fac678e3 export resourceGroup="poc-rg-ci-02"

    echo "Using servicePrincipalName $servicePrincipalName" echo "$roleName" echo "Using subscription ID $subscriptionID" echo "resourceGroup $resourceGroup" echo "Creating SP for RBAC with name $servicePrincipalName, with role $roleName and in scopes /subscriptions/$subscriptionID/resourceGroups/$resourceGroup" az ad sp create-for-rbac --name $servicePrincipalName --role $roleName --scopes /subscriptions/$subscriptionID/resourceGroups/$resourceGroup

  1. I have granted this service principal the Contributor role to the subscription.

    This will enable the service principal to create any azure resource. As a security best practice, you should adopt the least privilege principle by providing specific Azure AD RBAC roles.

  2. You need a service connection that the pipeline will refer to so that various pipeline tasks can execute.
    In project settings > Service Connections

  1. I have prepared these terraform files which are used by the pipeline.

Before developing the pipeline yaml file, since I am running terraform tasks, I need to install the Visual Studio Extension from the marketplace found at

https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks

Here are the terraform file contents:

Provision Azure Backend

Terraform needs to keep a State file to keep track of what Resources are managed by Terraform.
Using this State file, Terraform knows which Resources are going to be created/updated/destroyed by looking at your Terraform plan/template (we will create this plan in the next section).

So in Azure, we need a:

  • Storage Account:
    Create a Storage Account, any type will do, as long it can host Blob Containers.

  • Blob Container:
    In the Storage Account we just created, we need to create a Blob Container — not to be confused with a Docker Container, a Blob Container is more like a folder.

providers.tf

terraform {
  required_version = ">=1.0" # https://github.com/hashicorp/terraform/releases

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0" # https://registry.terraform.io/providers/hashicorp/azurerm/latest
    }
    random = {
      source  = "hashicorp/random"
      version = "~>3.0"
    }
  }


  backend "azurerm" {
    storage_account_name = "azteradevopsteraaz"
    container_name       = "tfstate"
    key                  = "terraform.tfstate"
    access_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx..."
  }

}

provider "azurerm" {
  subscription_id = var.subscription_id
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
  features {}
}

main.tf

data "azurerm_resource_group" "rg" {
  name = var.resource_group_name
}


resource "azurerm_storage_account" "example" {
  name                     = var.functionapp_storage_account_name
  resource_group_name      = data.azurerm_resource_group.rg.name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "example" {
  name                = "appserviceplan01"
  resource_group_name = data.azurerm_resource_group.rg.name
  location            = var.location
  os_type             = "Windows"
  sku_name            = "Y1"
}

resource "azurerm_windows_function_app" "example" {
  name                = var.azurerm_windows_function_app_name
  resource_group_name = data.azurerm_resource_group.rg.name
  location            = var.location

  storage_account_name       = azurerm_storage_account.example.name
  storage_account_access_key = azurerm_storage_account.example.primary_access_key
  service_plan_id            = azurerm_service_plan.example.id

  site_config {}
}

variables.tf

variable "environment" {
  type = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
  default = "centralindia"
}

variable "functionapp_storage_account_name" {
  type = string
}

variable "azurerm_windows_function_app_name" {
  type = string
}

variable "subscription_id" {
    type = string
    default = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

variable "client_id" {
    type = string
    default = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

variable "client_secret" {
    type = string
    default = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

variable "tenant_id" {
    type = string
    default = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

  1. I have attached the code of the YAML pipeline which is going to create the pipeline for us.

azure-pipelines.yml

trigger:
- main

pool:
  vmImage: ubuntu-latest

variables:
  - name: environment
    value: dev
  - name: location
    value: centralindia
  - name: subscriptionId
    value: ee22c8f0-93df-4b47-925a-d337fef522fe
  - name: serviceConnectionName
    value: AzTera
  - name: resource_group_name
    value: poc-rg-ci-02 
  - name: storage_account_name
    value: azteradevopsteraaz
  - name: functionapp_storage_account_name
    value: azteraterasteraaz
  - name: azurerm_windows_function_app_name
    value: azureterafunction


steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'


- task: TerraformInstaller@0
  displayName: install terraform
  inputs:
    terraformVersion: latest

- task: AzureCLI@2
  displayName: 'Create resource group $(resource_group_name)'
  condition: true
  inputs:
    azureSubscription: $(serviceConnectionName)
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az group create -n $(resource_group_name) -l $(location)


- task: AzureCLI@2
  displayName: 'Create storage account $(storage_account_name) for terraform state files'
  condition: true
  inputs:
    azureSubscription: $(serviceConnectionName)
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az storage account create -n $(storage_account_name) -g $(resource_group_name) -l $(location) --sku Standard_LRS


- task: TerraformTaskV2@2
  displayName: 'Terraform init'
  inputs:
    command: 'init'
    provider: 'azurerm'
    workingDirectory: '$(System.DefaultWorkingDirectory)/MainDir'
    backendServiceArm: '$(serviceConnectionName)'
    backendAzureRmResourceGroupName: '$(resource_group_name)'
    backendAzureRmResourceGroupLocation: '$(location)'
    backendAzureRmStorageAccountName: '$(storage_account_name)'
    backendAzureRmContainerName: 'tfstate'
    backendAzureRmKey: 'terraform.tfstate'
    commandOptions: '-lock=false'

- task: TerraformTaskV3@0
  displayName: 'Terraform plan'
  inputs:
    command: 'plan'
    workingDirectory: '$(System.DefaultWorkingDirectory)/MainDir'
    environmentServiceNameAzureRM: '$(serviceConnectionName)'
    commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)"  -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'


- task: TerraformTaskV3@0
  displayName: 'Terraform apply'
  inputs:
    command: 'apply'
    workingDirectory: '$(System.DefaultWorkingDirectory)/MainDir'
    environmentServiceNameAzureRM: '$(serviceConnectionName)' 
    commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)" -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'


- task: TerraformTaskV3@0
  displayName: 'Terraform destroy'
  condition: false # disable destroying
  inputs:
    command: 'destroy'
    workingDirectory: '$(System.DefaultWorkingDirectory)/MainDir'
    environmentServiceNameAzureRM: '$(serviceConnectionName)' 
    commandOptions: '-var "environment=$(environment)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)  -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" "'

Run the pipeline and we will get this result:

This is the storage container and the terraform state file in the storage account container that stores the current state of an infrastructure managed by Terraform.

Here’s a screenshot of the content available in the terraform.tfstate file

Here are the resources created by our pipeline.

Troubleshooting

Here are some errors that you may have encountered while trying to set up this demo.

Error: ##[error]Error: Input required: backendServiceArm

Solution: provide all backend* inputs

Error: ##[error]Error: There was an error when attempting to execute the process ‘/usr/local/bin/terraform’. This may indicate the process failed to start.

Solution: Make sure your paths are correct. Note that in Linux you have to use front slashes ‘/’

Error: “features”: required field is not set

Solution: This happened when I copied an existing Template from the Interwebs. Apparently, you need to specify a features {} key in the provider block (I’ve included it in my example)

Resources

Azure DevOps

Azure DevOps Services | Microsoft Azure

Terraform

Terraform by HashiCorp

AzureTerraformterraform-stateDevopsPipeline