Background
Managing the life cycle of PowerShell module assets in your Azure Automation accounts can be challenging. If you are currently using Azure Automation, you may have already noticed the following behaviours when managing the module assets:
1. It is difficult to automate the module asset deployment process.
If you want to automate the module deployment to your Automation Account (i.e. using the PowerShell cmdlet New-AzureRmAutomationModule), you must ensure the module that you are trying to import is zipped into a zip file and located on a public location where Azure Automation can read via HTTP (i.e. Azure Blob storage). In my opinion, this is over complicated.
2. Modules are not deployed to the Hybrid Workers automatically
If you are using Hybrid Workers, you must also manage the modules separately. Unlike Azure runbook workers, Azure Automation does not automatically deploy modules to Hybrid Workers. This means when you import a module to your Azure Automation account, you must also manually deploy it to your Hybrid Worker computers.
3. Difficult to maintain module version consistencies.
Since managing modules in your Azure Automation accounts and hybrid workers are two separate processes, it is hard to make sure the versions of your module assets are consistent between your automation account and hybrid worker computers.
Over the past few months, I have invested lot of my time on MyGet and looking for ways to close these gaps. Few months ago, I have released a PowerShell DSC Resource module called cPowerShellPackageManagement (https://blog.tyang.org/2016/09/15/powershell-dsc-resource-for-managing-repositories-and-modules/). By using this DSC resource module, we can easily develop DSC configurations for computers (such as Hybrid Workers) to automatically install modules from a PowerShell module repository (i.e. a MyGet feed). This approach closes the gaps of managing Hybrid Worker computers (item #2 on the list above). Today, I am going to discuss how we can tackle item #1 and #3. Before I start talking about my solutions, let me quickly introduce MyGet first.
What is MyGet?
Myget (www.myget.org) is a SaaS based package repository hosted on the cloud. It supports all the popular package providers such as NuGet, Npm etc. It can host both private and public repository (called a feed) for you or your organisation.
If you come from a developer or DevOps background, you may have already heard about MyGet in the past, or have used similar on-premises package repositories (such as ProGet). If you are an IT Pro, since you are reading this blog post right now, you must be familiar with PowerShell, therefore must have heard or used PowerShell Gallery (https://powershellgallery.com). You can use MyGet the same way as PowerShell Gallery in PowerShell version 5 and later, except you have absolute control of the content in your feeds. Also, if you are using a paid MyGet account, you can have private feeds and you can control the access by issuing API keys. You can also create multiple feeds that contain different packages (PowerShell modules in this case). i.e. if you develop PowerShell modules, you can have a Dev feed for you to use during development, and also Test and Production feeds for testing and production uses.
Why Do I Need MyGet?
You may be a little bit hesitate to use PowerShell Gallery because it is 100% public. As a regular user like everyone else, you can only do very little. i.e. you can publish modules to PowerShell gallery, but you can’t guarantee your modules will stay there forever. Microsoft may decide to un-list your modules if they find problems with it (i.e. failed to comply with the rules set in the PSScriptAnalyzer). You also don’t have access to delete your modules from PowerShell Gallery. You can un-list your modules, but they are still hosted there. To me, PowerShell Gallery is more like a community platform that allows everyone to share their work, but you should not use it in any production environments because you don’t have any controls on the content, how can you make sure the content you need is going to be there tomorrow?
MyGet allows you to create feeds that you have total control, and as I mentioned already, with a paid MyGet account, you can have private feeds to host your IPs that you don’t want to share with the rest of the world.
MyGet also ships with other awesome features, such as Webhook support.
Automating Module Deployment to Automation Account
I have developed a runbook that retrieves a list of modules from a repository (i.e. your MyGet feed), and import each module to the Automation account of where the runbook resides, if the module does not exist or the version is lower than the latest available version from the module repository. Before importing, the runbook also tries to work out the module dependencies and import required modules in groups (i.e. the modules without dependencies are imported first). Here’s the runbook source code:
https://gist.github.com/tyconsulting/df78d43e64fe86fe772f947fded3c4da
Note: this runbook does not download and zip up PowerShell modules from the repository feed. Instead, it construct the URI to the underlying NuGet package and import the package directly to your automation account.
In order to use the runbook, you will need to create a automation variable first.
Name: ModuleFeedLocation
Value: **
Note: if you are not sure what is the source location URI for your feed, check out this help document from MyGet website: http://docs.myget.org/docs/how-to/publish-a-powershell-module-to-myget. However, I don’t believe the documentation is 100% accurate. Based on my experience, no matter if you are using private or public feeds, the Source location URI should be:
https://www.myget.org/F/
The API key is available on the MyGet portal:
if you have connected to the feed as a PowerShell repository, you can also check using Get-PSRepository cmdlet:
Other than the automation variable, you will also need to make sure you have the AzureRunAsConnection connection asset and associated certificate asset created. these assets are created automatically by default when you created your Azure Automation account:
If you don’t have this connection asset, you can manually create it using PowerShell – this process is documented here: https://docs.microsoft.com/en-au/azure/automation/automation-sec-configure-azure-runas-account
Once the runbook and all required assets are in place, you will also need to create a webhook for the runbook. It is OK to configure the webhook to target Azure workers (although targeting hybrid worker group is also OK, but not necessary).
Once the webhook is created, go to MyGet portal, go to your feed then go to the Webhook section and add a HTTP POST webhook
then enter a description and paste the runbook webhook URL. for the webhook trigger, only tick “Package Added”:
Once the webhook trigger is created, everything is good to go. when next time you add a PowerShell module or update an existing module on your MyGet feed, it will automatically trigger the Azure Automation runbook, which will find the modules need to be imported and updated, and attempt to import them one a time.
Tips:
Once you have configured your MyGet feed as a PowerShell repository on a computer running PowerShell v 5 or later, you can publish modules located on your local computer to the feed using Publish-Module cmdlet. You can also configure MyGet to get modules from another repository such as PowerShell Gallery. I have blogged this previously: https://blog.tyang.org/2016/09/20/pushing-powershell-modules-from-powershell-gallery-to-your-myget-feeds-directly/
If you want to configure multiple Automation accounts to sync with a single MyGet feed, you can simply create the runbook and required assets in each automation account, and add a webhook trigger for each instance of the runbook within your MyGet feed.
Things to Watchout
there are few things that you need to watch out when using this solution:
1. be aware of the limitations in Azure Automation
Some of these limitations may impact your module imports. you can find the official documentation here: https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#automation-limits
2. Unlike any NuGet repositories such as PowerShell Gallery and MyGet, Azure Automation does not support storing different versions of same module
This may cause some of the module imports to fail. For example, if you have a module called ModuleA (version 1.0) that is a dependency to ModuleB version 1.0. You have ModuleA 1.0, ModuleB 1.0 and 2.0 in your MyGet repository, the runbook will firstly import ModuleB 2.0 to your automation account first. then when it tries to import ModuleA 1.0, it may fail because it does not pass the validation test (by importing ModuleA 1.0 on the runbook worker computer). so prior to committing these kind of packages to a feed that’s being used by Azure Automation, make sure you test it first on another feed, and make sure you can successfully install and import the module on your local computer.
3. Do not load too many modules to the feed initially
Module import into Azure Automation account takes a lot of time. when running a runbook job on Azure workers, the runbook can run maximum 3 hours due to its fair share policy. so if you have a lot of modules to load in the beginning, you need to make sure the runbook job can be completed within 3 hours. or you may have to rerun the runbook to pickup the modules didn’t get imported in the previous runbook job. Alternatively, you can configure the runbook to run on a Hybrid Worker group, because the fair share policy does not apply when the job is being executed on hybrid workers.
Conclusion
If you use a dedicated MyGet feed to host all required modules for Azure Automation, you can use the cPowerShellPackageManagement DSC resource module I mentioned earlier in this blog post to automate the module deployment to Hybrid Workers. In the same time, by using the method described in this blog post, you have also got the Automation account covered.
Therefore, if you have both DSC configured for Hybrid Workers (i.e using Azure Automation DSC), and have this runbook and webhook configured, by adding a new package to your MyGet feed, your entire Azure Automation infrastructure is updated automatically.
My MVP buddy Alex Verkinderen also also done some interesting integration between MyGet and PowerShell Gallery. He is going to publish his innovation on his blog (http://www.mscloud.be/) soon, so make sure you subscribe to his blog 
.
Lastly, thanks Alex for testing the runbook for me, and if anyone has any questions or suggestions, please feel free to contact me.
Leave a comment