Tag Archives: MimboloveAzure Functions

Deploying ARM Templates with Artifacts Located in a Private GitHub Repository

Written by Tao Yang

Background

I have spent the last few days authoring an Azure Resource Manager (ARM) template. The template is stored in a private GitHub repository. It contains several nested templates, one of which deploys an Azure Automation account with several runbooks. For the nested templates and automation runbooks, the location must be a URI. Therefore the nested templates and the Azure Automation runbooks that I wish to deploy in the ARM templates must be located in location that is accessible by Azure Resource Manager. There are many good examples in the Azure Quickstart Template GitHub repository, for example, in the oms-all-deploy template, these artifacts are being referenced as URIs pointing to the raw content in GitHub:

Nested Templates:

image

image

Azure Automation runbook:

image

As you can see from the examples above, the template is referencing these artifacts as the raw file content in github (located in https://raw.githubusercontent.com). This method works well when your templates and artifacts are stored in a public GitHub repository, but it won’t work when they are stored in a private repository. For example, if I click on the “Raw” button for a file stored in a private repo like shown below:

image

you will see a token was added to the raw file content URI:

image

This token was generated for your current session once you’ve logged in to GitHub. Although as most of you guys already know that you can generate Personal Access token in GitHub, you can not replace this token in the URL with the Personal Access Token you have generated. The personal access token must be added as part of the authorization header in the HTTP request, and you also need to add another header “Accept”: “application/vnd.github.VERSION.raw”. Obviously we cannot add HTTP headers in ARM templates, therefore we cannot reference artifacts located in a private GitHub repositories out of the box. Someone has already started a thread in the MSDN forum: https://social.msdn.microsoft.com/Forums/sqlserver/en-US/54876f00-4bf4-4d84-81af-e7a1128ac6f7/linking-arm-templates-in-private-github-repo?forum=windowsazurewebsitespreview. the workaround they came up with was building a HTTP proxy server and add the required headers in the proxy server. This to me seems awfully complicated and it also brings limitations that you can only deploy the templates in the network that you can access such a proxy server.

Solution

I spent most of my day yesterday trying to figure out if there is a way to deploy artifacts located in a private GitHub repository and I couldn’t find a way to use the Personal Access Token as a parameter in the HTTP GET request. As I accepted the fact that I wouldn’t be able to do it and started removing all nested templates in my project, I had a light bulb moment when I was driving to the supermarket yesterday afternoon, and the idea I came up with is super simple:  since I cannot pass the personal access token in the URL, I can just write a very simple “proxy” Azure Function app that accept the GitHub personal access token from the URL parameter, then construct the HTTP header, make the request to GitHub, and return the HTTP response it received from GitHub. Once this function is written, we can use the URI to the function as the artifact location instead of the GitHub URI.

GitHubPrivateRepoFileFetcher Azure Function

I’ve written something similar before, so this HTTP Trigger C# Azure function app literally took me 10 minutes to write. I named this function GitHubPrivateRepoFileFetcher:

This function requires 2 parameters to be passed in from the URL:

  • githuburi – the original URI to the raw GitHub file content
  • githubaccesstoken – the GitHub personal access token you have generated

Since the function does not contain any sensitive information, and I’d like to ensure the function URI is not too long, I have configured the function Authorization level to Anonymous so I don’t have to use an authorization code to invoke the Azure Function.

image

To call this function, you need to construct the URL as the following:

https://<Function App Name>.azurewebsites.net/api/GitHubPrivateRepoFileFecher?githuburi=https://raw.githubusercontent.com/<GitHub User Name>/<Repository>/<branch>/<path to the file>&githubaccesstoken=<GitHub Person Access Token>

i.e.

https://myfunctionapp.azurewebsites.net/api/GitHubPrivateRepoFileFecher?githuburi=https://raw.githubusercontent.com/tyconsulting/TestPrivateRepo/master/DemoNestedTemplates/azuredeploy.json&githubaccesstoken=e82dc3df60b92147c81a9924042da8d7f0bc78c8

GitHub Personal Access Token

Before start using the GitHubPrivateRepoFileFetcher function, you will firstly need to generate a GitHub personal access token if you don’t already have one. The token must have access to repo as shown below:

image

Constructing ARM Templates

In the ARM templates, define the following parameters (and define the default value to match your environment):

define the linked template URL in the variables section:

For the Azure Automation runbook that I am planning to deploy, define the URI to the script in the variables section similar to the previous example:

For the Azure Automation runbook resource, use the variable defined above in the script uri property:

GitHub README.md

Most of the ARM template repositories contains the “Deploy to Azure” and “Visualize” buttons as shown below:

image

To add the “Deploy to Azure” button to your README.md markdown file, you will need to add the following code to the markdown (modify the URL to suit your environment):

For the “Visualize” button, add the following code in the markdown:

Basically, add the Azure Function URL with required parameters as the parameter to the azure custom deployment and armviz.io URIs. you will need to encode the function URI. you can use an an online encoder utility such as http://www.url-encode-decode.com/ to encode the original Azure Function URI.

When you click the “Deploy to Azure” button, you will be redirected to the Azure portal:

image

When the deployment is completed, the resources defined in the ARM templates are created:

image

Conclusion

By using the “proxy” Azure Function GitHubPrivateRepoFileFetcher, you can easily retrieve the content of a file in private GitHub repo without having to use custom headers in HTTP request. This “proxy” Azure Function is generic that you can use for any Azure subscriptions and any GitHub private repositories. If you regularly use GitHub private repositories for ARM templates, I strongly recommend you to create such a Azure function to assist you with your deployments.

If you have any questions or recommendations, please feel free to contact me.

Feeding Your Power BI Reports from Azure Functions

Written by Tao Yang

Background

Few days ago my good friend and fellow CDM MVP Alex Verkinderen (@AlexVerkinderen) had a requirement to produce a Power BI dashboard for Azure AD users. so Alex and I started discussing a way to produce such report in Power BI. After exploring various potential possibilities, we have decided to leverage Azure Functions to feed data into Power BI. You can check out the Power BI solution Alex has built on his blog here: http://www.mscloud.be/retrieve-azure-aad-user-information-with-azure-functions-and-publish-it-into-powerbi

In this blog post, I’m not going to the details of how the AAD Users Power BI report was built. Instead, I will focus on the Azure Functions component and briefly demonstrate how to build a Azure Functions web service and act as a Power BI data source. As an example for this post, I’ll build a Azure Functions web service in PowerShell that brings in Azure VMs information into Power BI. To set the stage, I have already written two blog posts yesterday on Azure Functions:

These two posts demonstrated two important steps that we need to prepare for the Azure Functions PowerShell code. We will need to follow these posts and prepare the following:

  • Upload the latest AzureRM.Profile and AzureRM.Compute PowerShell modules to Azure Functions
  • Encrypt the password for the service account to be used to access the Azure subscription.

Once done, we need to update the user name and the encrypted password in the code below (line 24 and 25)

I have configured the function authorization level to “Function” which means I need to pass an API key when invoking the  function. I also need to pass the Azure subscription Id via the URL. To test, I’m using the Invoke-WebRequest cmdlet and see if I can retrieve the Azure VMs information:

As you can see, the request body content contains a HTML output which contains a table for the Azure VM information

image

Now that I’ve confirmed the function is working, all I need to do is to use Power BI to get the data from the web.

Note: I’m not going to too deep in Power BI in this post, therefore I will only demonstrate how to do so in Power BI desktop. However Alex’s post has covered how to configure such reports in Power BI Online and ensuring the data is always up-to-date by leveraging the On-Prem Data Gateway component. So, please make sure you also read Alex’s post when you are done with this one.

image

In Power BI Desktop, simply enter the URL with the basic setting:

image

and choose “Table 0”:

image

Once imported, you can see the all the properties I’ve defined in the Azure Functions PowerShell script has been imported in the dataset:

image

and I’ve used a table visual in the Power BI report and listed all the fields from the dataset:

image

Since the purpose of this post is only to demonstrate how to use Azure Functions as the data source for Power BI, I am only going to demonstrate how to get the data into Power BI. Creating fancy reports and dashbaords for Azure VM data is not what I intent to cover.

Now that the data is available in Power BI, you can be creative and design fancy reports using different Power BI visuals.

Note: The method described in this post may not work when you want to refresh your data after published your report to Power BI Online. You may need to use this C# Wrapper function: http://blog.tyang.org/2016/10/13/making-powershell-based-azure-functions-to-produce-html-outputs/. Alex has got this part covered in his post.

Lastly, make sure you go check out Alex’s post on how he created the AAD Users report using this method. As I mentioned, he has also covered two important aspects – how to make this report online (so you can share with other people) and how to make sure you data is always up to date by using the on-prem data gateway.

Making PowerShell Based Azure Functions to Produce HTML Outputs

Written by Tao Yang

Over the last few weeks, I’ve been working with my MVP buddy Alex Verkinderen (@AlexVerkinderen) on some Azure Function related stuff. We have both written few PowerShell based functions that output a HTML page.

These functions use the ConvertTo-HTML cmdlet to produce the HTML output. For example, here’s a simple one that  list 2 cars in a HTML table:

Today we ran into an issue while preparing for our next blog posts, after some diagnostics, we realised the issue was caused by the HTML output returned from the PowerShell based functions.

If I use Invoke-WebRequest cmdlet in Powershell to trigger this PowerShell function, I am able to get the HTML output in the request output content and everything looks good:

image

However, if we simply invoke this function from a browser, although the output is in HTML format, the browser does not display the HTML page. it displays the HTML source code instead:

image

after some research, we found the cause of this issue – the content type returned by the PowerShell function is always set to “text/plain”:

image

I suspect this is because for PowerShell based functions, we have to output to a file ($res variable by default). I have tried to construct a proper HTTP response message (System.Net.Http.HttpResponseMessage), but it didn’t work in the PowerShell functions. Based on my testing results, it seems PowerShell functions cannot handle complex types.

Luckily I found this post and it pointed me to the right direction: http://anthonychu.ca/post/azure-functions-serve-html/. According on this post, we can certainly serve out a proper HTML page in C# based functions.

I don’t really want to rewrite all my PowerShell functions to C#, not only because I don’t want to reinvent the wheels, but also I want to keep using the PowerShell modules in those existing functions. In the end, I came up with C# based “wrapper” function. I named this function HTTPTriggerProxy:

This C# based HTTPTriggerProxy function simply takes the URL you have specified, get the response and wrap it in a proper HTTPResponseMessage object. All you need to do is to specify the original URL that you want to request in the “RequestURL” parameter as part of the wrapper function URL:

https://<Your Azure Function Account>.azurewebsites.net/api/HttpTriggerProxy?code=<Access code for Http Trigger Proxy function>&RequestURL=<Your original request URL>.

Now if I use this wrapper to invoke the sample GetCars PowerShell function, the HTML page is displayed in the browser as expected:

image

and you can see the content type is now set as “text/html”:

image

Note:

  • This wrapper function only supports the Get HTTP method. The Post method is not supported so you can only pass the RequestURL in the the wrapper URL (as opposed to placing it in the request body). I didn’t bother to cater the POST method in this function because what we are going to use this for only supports HTTP Get method.
  • if your original request requires authentication, then this is not going to work for you.
  • If you original URL contains the ampersand character (“&”), please replace it with “%26”. for example, if your original request is https://myazurefunction.azurewebsites.net/api/GetCars?code=rgpxmm0p87fh2z1wd0a6vargfxxogb6cf&colour=red, then you need to change it to https://myazurefunction.azurewebsites.net/api/GetCars?code=rgpxmm0p87fh2z1wd0a6vargfxxogb6cf%26colour=red

Lastly, this is just something we came up today while making another set of posts. Please stay turned. our new posts will be published in the next day or two.

Securing Passwords in Azure Functions

Written by Tao Yang

09/10/2016 – Note: This post has been updated as per David O’Brien’s suggestion .

As I mentioned in my last post, I have started playing with Azure Functions few weeks ago and I’ve already built few pretty cool solutions. One thing that I’ve spent a lot of time doing research on is how to secure credentials in Azure Functions.

Obviously, Azure Key Vault would be an ideal candidate for storing credentials for Azure services. If I’m using another automation product that I’m quite familiar with – Azure Automation, I’d certainly go down the Key Vault path because Since Azure Automation account already creates a Service Principal for logging into Azure and we can simply grant the Azure AD Application access to the Key Vault. However, and please do point me to the correct direction if I’m wrong, I don’t think there is an easy way to access the Key Vault from Azure Functions at this stage.

I cam across 2 feature requests on both Github and UserVoice suggesting a way to access Key Vault from Azure Functions, so I hope this capability will be added at later stage. But for now, I’ve come up a simple way to encrypt the password in the Azure Functions code so it is not stored in clear text. I purposely want to keep the solution as simple as possible because one of the big advantage of using Azure Functions is being really quick, therefore I believe the less code I have to write the better. I’ll use a PowerShell example to explain what I have done.

I needed to write a function to retrieve Azure VMs from a subscription – and I’ll blog the complete solution next time. Sticking with the language that I know the best, I’m using PowerShell. I have already explained how to use custom PowerShell modules in my last post. In order to retrieve the Azure VMs information, we need two modules:

  • AzureRM.Profile
  • AzureRM.Compute

I use the method explained in the previous post and uploaded the two modules to the function folder. Obviously, I also need to use a credential to sign in to my Azure subscription before retrieving the Azure VM information.

I’m using a key (a byte array) to encrypt the password secure string.  If you are not familiar with this practice, I found a very detailed 2-part blog post on this topic, you can read them here:

Secure Password With PowerShell: Encrypting Credentials – Part 1

Secure Password With PowerShell: Encrypting Credentials – Part 2

So firstly, I’ll need to create a key and store the content to a file:

I then uploaded the key to the Azure Functions folder – I’ve already uploaded the PowerShell modules to the “bin” folder, I created a sub-folder under “bin” called Keys:

image

I wrote a little PowerShell function (that runs on my PC, where a copy of the key file is stored) to encrypt the password.

PowerShell function Get-EncryptedPassword:

I call this function to encrypt the password and copy the encrypted string to the clipboard:

image

I then created two app settings in Azure Functions Application settings:

  • AzureCredUserName
  • AzureCredPassword

The AzureCredUserName has the value of the user name of the service account and AzureCredPassword is the encrypted string that we prepared in the previous step.

image

I then paste the encrypted password string to my Azure Functions code (line 24):

The app settings are exposed to the Azure functions as environment variables, so we can reference them in the script as $env:AzureCredUserName and $env:AzureCredPassword (line 23 and 24)

image

As shown above, to decrypt the password from the encrypted string to the SecureString, the PowerShell code reads the content of the key file and use it as the key to convert the encrypted password to the SecureString (line 26-27). After the password has been converted to the SecureString, we can then create a PSCredential object and use it to login to Azure (line 28-29).

Note: If you read my last post, I have explained how to use Kudu console to find the absolute path of a file, so in this case, the file path of the key file is specified on line 26.

Needless to say, the key file you’ve created must be stored securely. For example, I’m using KeePass to store my passwords, and I’m storing this file in KeePass. Do not leave it in an unsecured location (such as C:\temp as I demonstrated in this example).

Also, Since the app settings apply to all functions in your Azure Functions account, you may consider using different encryption keys in different functions if you want to limit which which function can access a particular encrypted password.

Lastly, as I stated earlier, I wanted to keep the solution as simple as possible. If you know better ways to secure passwords, please do contact me and I’d like to learn from you.

Using Custom PowerShell Modules in Azure Functions

Written by Tao Yang

Like many other fellow MVPs, I have started playing with Azure Functions over the last few weeks. Although Azure Functions are primarily designed for developers and supports languages such as C#, Node.JS, PHP, etc. PowerShell support is currently in preview. This opens a lot of opportunities for IT Pros. My friend and fellow CDM MVP David O’Brien has written some really good posts on PowerShell in Azure Functions (https://david-obrien.net/). Although the PowerShell runtime in Azure Functions comes with a lot of Azure PowerShell modules by default (refer to David’s post here for details), these modules are out-dated, and some times, we do need to leverage other custom modules that are not shipped by default.

While I was trying to figure out a way to import custom modules into my PowerShell Azure Functions, I came across this post showing me how to upload 3rd party assemblies for C# functions: http://www.robfox.nl/2016/04/27/referencing-external-assemblies-azure-functions/. So basically for adding assemblies for C#, you will need to create a folder called “bin” under your function root folder, and upload the DLL to the newly created folder using a FTP client. I thought I’d give this a try for PowerShell modules, and guess what? it worked! I’ll use one of my frequently used module called GAC as an example in this post and work through the process of how to prepare the module and how to use it in my PowerShell code.

01. I firstly download the Gac module from the PowerShell Gallery (https://www.powershellgallery.com/packages/Gac/1.0.1):

02. Make sure the Azure Functions App Service has the deployment credential configured

image

03. FTP to the App Service using the deployment credential configured in the preview step, create a “bin” folder under the Azure Functions folder (“/site/wwwroot/<Azure Functions Name>”) and upload the module folder:

image

04. In Azure Functions, launch the Kudu console

image

05. Identify the PowerShell module file system path in Kudu. The path is D:\home\site\wwwroot\<Azure Function Name>\bin\<PS module name>\<PS module version>

image

06. By default, the PowerShell runtime is configured to run on 32-bit platform. If the custom module requires 64-bit platform, you will need to configure the app setting and set the Platform to 64-bit

image

image

Now that the module is uploaded, and because the module is not located in a folder that’s listed in the PSModulePath environment variable, we have to explicitly import the module manifest (.psd1 file) before using it. For example, I have created a function with only 2 lines of code as shown below:

The “Get-GacAssembly” cmdlet comes from the Gac PowerShell module. As the name suggests, it lists all the assemblies located in the Gac (Global Assemblies Cache). When I call the HTTP trigger function using Invoke-WebRequest, you’ll see the assemblies listed in the logs window:

image

image

I have also tested stopping and restarting the Azure App Service, and I can confirm the module files stayed at the original location after the restart based on my tests.

This concludes my topic for today. I have few other really cool blogs in the pipeline for using PowerShell in Azure Functions, stay tuned.