Pester Test Your ARM Template in Azure DevOps CI Pipelines

Introduction

It is fair to say, I have spent a lot of time on Pester lately. I just finished up a 12 months engagement with a financial institute here in Melbourne. During this engagement, everyone in the project team had to write tests for any patterns / pipelines they are developing. I once even wrote a standalone pipeline only to perform Pester tests. One of the scenario we had to cater for is: How can you ensure the ARM template you are deploying only deploys the resources that you intended to deploy? In another word, if someone has gone rogue or mistakenly modified the template, how can you make sure it does not deploy resources that’s not supposed to be deployed (i.e. a wide open VNet without NSG rules).

To cater for this requirement, an engineer from the customer’s own cloud team has written a Pester test that validates the content of the ARM templates by parsing the JSON file. I like the idea, but since I didn’t bother (and couldn’t) keep a copy of the code, I wrote my own version, with some improvements and additional capability. The pester test I wrote performs the following tests:

• Template file validation
  • Test if the ARM template file exists
  • Test if the ARM template is a valid JSON file
• Template content validation
  • Contains all required elements (defined by Microsoft’s ARM template schema)
  • Only contains valid elements
  • Has valid Content Version
  • Only has approved parameters
  • Only has approved variables
  • Only has approved functions
  • Only has approved resources
  • Only has approved outputs

For example, say I have a template that deploys a single VM. This template has the following elements defined:

• parameters:
  • virtualMachineNamePrefix
  • virtualMachineSize
  • adminUserName
  • virtualNetworkResourceGroup
  • virtualNetworkName
  • adminPassword
  • subnetName
• variables:
  • nicName
  • publicIpAdressName
  • publicIpAddressSku
  • publicIpAddressType
  • subnetRef
  • virtualMachineName
  • vnetId
• functions:
  • namespace: tyang, member: uniqueName
• resources (of it’s type):
  • Microsoft.Compute/virtualMachines
  • Microsoft.Network/networkInterfaces
  • Microsoft.Network/publicIpAddresses
• outputs:
  • adminUserName

Using this Pester test script, I can either be very strict and ensure ALL the elements listed above must be defined (and nothing else), or be less restrictive, only test against the required element (resources) and one or more optional elements (parameters, variables, functions and outputs).

Test.ARMTemplate.ps1

Here’s the code, hosted on GitHub:

To test your template using this script, you will need to pass the following parameters in:

-TemplatePath

The path to the ARM Template that needs to be tested against.

-parameters

The names of all parameters the ARM template should contain (optional).

-variables

The names of all variables the ARM template should contain (optional).

-functions

The list of all the functions (namespace.member) the ARM template should contain (optional).

-resources

The list of resources (of its type) the ARM template should contain. Only top level resources are supported. child resources defined in the templates are not supported.

-output

The names of all outputs the ARM template should contain (optional).

Examples:

Test ARM template file with parameters, variables, functions, resources and outputs:

$params = @{
TemplatePath = 'c:\temp\azuredeploy.json'
parameters = 'virtualMachineNamePrefix', 'virtualMachineSize', 'adminUsername', 'virtualNetworkResourceGroup', 'virtualNetworkName', 'adminPassword', 'subnetName'
variables = 'nicName', 'publicIpAddressName', 'publicIpAddressSku', 'publicIpAddressType', 'subnetRef', 'virtualMachineName', 'vnetId'
functions = 'tyang.uniqueName'
resources = 'Microsoft.Compute/virtualMachines', 'Microsoft.Network/networkInterfaces', 'Microsoft.Network/publicIpAddresses'
outputs = 'adminUsername'
}
.\Test.ARMTemplate.ps1 @params

Test ARM template file with only the resources elements:

$params = @{
TemplatePath = 'c:\temp\azuredeploy.json'
resources = 'Microsoft.Compute/virtualMachines', 'Microsoft.Network/networkInterfaces', 'Microsoft.Network/publicIpAddresses'
}
.\Test.ARMTemplate.ps1 @params

Using it in Azure DevOps Pipeline

To use this Pester test script in your VSTS pipeline, you can follow the steps listed below:

1. Include this script in your repository

In my Git repo, I have created a folder called “tests” and placed this script inside a sub-folder of “tests”:

2. Create a variable group and define the input parameters for the pester test script in the variable group.

In this demo, I name these variables “parameters”, “variables”, “functions”, “resources” and “outputs”. these variables can be arrays, you can separate each value using comma “,”.

Please note that in addition to the template file path, only “resources” is the required parameter, if you don’t want to validate other template elements, you don’t need to create other variables here.

3. Link the variable group to the build pipeline

Before the build pipeline can use these variables, you need to link the variable group to the build pipeline.

4. In the build (CI) pipeline, add a “Pester Test Runner” task

Although there are other Pester test tasks out there, this is my favourite one.

5. Configure the Pester Test Runner task

Since the Pester test script requires input parameters, the “Script Folder or Script” field needs to be a hash table. In my demo, I’m leveraging the build-in variable $(System.DefaultWorkingDirectory) as well as my user-defined variables. It is set to:

@{Path=’$(System.DefaultWorkingDirectory)\ARMDeploymentDemo\SingleVMPatternDemo\tests\ARMTemplate\Test.ARMTemplate.ps1’; Parameters=@{TemplatePath =’$(System.DefaultWorkingDirectory)\ARMDeploymentDemo\SingleVMPatternDemo\azuredeploy.json’; parameters =$(parameters); variables = $(variables); functions = $(functions); resources = $(resources); outputs = $(outputs)}}

You will need to modify it to suit your needs.

6. Add a Publish Test Results task

Make sure the Test results files pattern matches what you configured as the Results file in the previous step.

7. Configure job dependencies

If you have any agent jobs after the Pester test, you can configure them to be depended on the Pester test job so they wont run if the Pester test has failed.

Accessing the test results

You can view the result of each individual test from the build logs:

You can also create a widget in a dashboard and view the test results in a graph:

Limitations

Initially, I also wanted to do a full-blown JSON schema validation by ad-hoc downloading ARM template schema, and validate the ARM template against the schema (which should be always up to date). I found this simple PowerShell script by the author of Newtonsoft.JSON libraries: https://gist.github.com/JamesNK/7e6d026c8b78c049bb1e1b6fb0ed85cf. It was a good starting point, but since the ARM template schema contains many $ref elements that reference other schemas, I had to modify this script to use the JsonSchemaResolver in order to resolve these references. Based on my experience, it was a hit and miss, not all references could be resolved because I was getting errors for some specific resource types defined in the ARM template, and after I spent almost a day trying to get this working, I got an error saying I have reached the hourly limit (https://www.newtonsoft.com/store/jsonschema) and I need to purchase a commercial license. This prompted me to stop exploring further because I don’t really want to develop a free solution that relies on commercial licenses. Therefore, this Pester test does not perform ARM JSON schema validation against your template. In my opinion, the best way to do it is to validate your template directly against the ARM engine (do a validation only deployment):

Although I have not used it myself, but I just want to point out, there is also a Pester test task available for ARM test deployments: https://marketplace.visualstudio.com/items?itemName=petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-Pester

Therefore, my script does not cover the full schema validation, but performing a test deployment would achieve the same result (and more).

If anyone knows how to validate JSON against a schema with online references for free, please ping me