This Concludes My Year 2014

Written by Tao Yang

It is one day away from the holiday season of the year. And I have worked HARD over the last few days so I can post my last technical post for the year 2014 before holidays.

First of all, I’d like to wish everyone a Merry Christmas and Happy New Year!

image

2014 has been a fantastic year for me. Here are some of the highlights for me in 2014:

I’ve been awarded as a Microsoft System Center Cloud and Data Center Management MVP for the first time in 1st July 2014.

This is truly my biggest accomplishment of the year. Not to mention being nominated by one of the most well known community leaders in System Center is an accomplishment by itself.

As part of a project team, the project team and I have successfully implemented one of the largest System Center 2012 infrastructures in the country (based on number of seats and number of System Center components implemented).

For those who knows me well, you probably know which one am I talking about Smile.

Had privilege and opportunity to attend Microsoft Global MVP Summit held in Redmond WA in November

I am so glad that I had the opportunity to attend such a wonderful event. Although pretty much everything is under NDA, I can’t really talk about the content of the sessions. I think I can share some pictures here (Some taken from the camera on my phone, some from the SLRs of other SCCDM MVPs).

image

image

image

image

image

image

image

Had opportunities to meet many big names (MVPs and Microsoft employees) in System Center during the MVP summit. Many of those have become good friends too.

I brought a lot of Tim Tam and Kangaroo Jerky to the summit. I didn’t expect Tim Tam to be so popular Smile. If I get awarded again in July 2015, I will make sure I’ll use a bigger suitcase for Tim Tam for the MVP Summit 2015.

image

image

Released a few new and updated Management Packs (ConfigMgr client MP, OpsMgr Self Maintenance MP, SCOM Maintenance Mode Scheduler MP, etc.), OpsMgr dashboards, PowerShell Scripts, SMA Modules etc. to the community.

I’ve lost count, but they should be all on this blog Smile.

Have written 63 blog posts (including this one) in total.

I don’t think the number is very high (only about 5 posts per month), but I’m trying my best Smile. Some of these posts are posting MPs, scripts etc. that I have spent a very long time on. Based on the content, I personally think this is quiet an achievement!

Clocked up over 170,000 hits on this blog in 2014 (to date).

Well, I think I still have a long way to go if comparing with some other popular System Center blogs (Not that I will turn this into a pissing contest). However t is a steady increase from 2013. But I’m sure I’ll do better next year.

What’s Next?

If everything goes as planned, this post will be my final words for 2014. I am taking some time off during the holiday seasons (well, not too long, going back to work on 5th Jan).

During my time off, I will probably spend few days working on an automation solution for OpsMgr – Something I’ve been working on during my spare time since August this year. This leads to the next point.

MVP ComCamp 2015

image

I have been chosen to speak at the MVP Community Camp 2015 in Melbourne on Friday 30th Jan 2015. I will be presenting the topic “Automating SCOM tasks using SMA”. This is something I’ve been working on since August this year. Personally, I think what I have done so far is really cool. This event is going to be held at the Microsoft Melbourne office at Freshwater Place, South Bank, Melbourne. Besides myself, a veteran MVP in System Center ConfigMgr, James Bannan will also deliver a session in Enterprise Mobility Suite in this event.  If you are based in Melbourne, please check out the detail of this event and sessions HERE. I am looking forward to speaking to the Melbourne based System Center folks Smile.

Lastly, I wish everyone have a wonderful time during this holiday season. I will be back in 2015 Smile.

A SMA Integration Module For SharePoint List Operations

Written by Tao Yang

Background

Many Microsoft System Center Orchestrator and Service Management Automation (SMA) users may agree with me, that these two automation platform does not have feature rich end user portals natively. Although System Center Service Manager can be used as a user portal for triggering SCORCH/SMA runbooks, Microsoft SharePoint is also a very good candidate for this purpose.

Integrating SharePoint with Orchestrator and SMA is not something new, many people have done this already. i.e.

System Center Universe America 2014 – Orchestrating Daily Tasks Like a Pro (by Pete Zerger and Anders Bengtsson)

Service Management Automation and SharePoint (by Christian Booth and Ryan Andorfer)

In my opinion, SharePoint (especially SharePoint lists) provides a quick and easy way to setup a web based end user portal for orchestration runbooks. I have also blogged my experiences in the past:

My Experience Manipulating MDT Database Using SMA, SCORCH and SharePoint

SMA Runbook: Update A SharePoint 2013 List Item

To me, not only I am using SharePoint 2013 in my lab; SharePoint Online from my Office 365 subscription, I also have no choice but using SharePoint 2010 in real life.

In my opinion, it is complicated to write SMA runbooks to interact with SharePoint (Using SharePoint web based APIs), not to mention the different versions of SharePoint also dictates how the runbook should be written. It is easier to use Orchestrator as a middle man in between SMA and SharePoint so we can use Orchestrator’s SharePoint integration pack.

Earlier this month, I was developing solutions to use SMA and Azure Automation to create OpsMgr Management Packs catalog on SharePoint 2013 / SharePoint Online sites. I have blogged the 2 solutions here:

On-Premise Solution (SMA + SharePoint 2013)

Cloud Based Solution (Azure Automation + SharePoint Online)

As I mentioned in the previous posts, I had to write a separate SMA module to be used in Azure Automation to interact with SharePoint Online because SharePoint Online sites require a different type of credential (SharePointOnlineCredential) that the PowerShell cmdlet Invoke-RESTMethod does not support. I called that module SharePointOnline back in the previous post and it utilises assemblies from the SharePoint Client Component SDK. I think the SharePoint people also refer to this SDK as Client-Side Object Model (CSOM)

After the MP catalogs posts were published, I have decided to spend a bit more time on the SharePoint Client Component SDK and see if it can help me simplify the activities between SMA and SharePoint. I was really happy to find out, the SharePoint Client Component SDK works for SharePoint 2013, SharePoint Online and SharePoint 2010 (limited). So I have decided to update and extend the original module, making it a generic module for all 3 flavours of SharePoint.

After couple of weeks of coding and testing, I’m pleased to announce the new module is now ready to be released. I have renamed this module to SharePointSDK (Sorry I’m not really creative with names Smile with tongue out).

 

SharePointSDK Module Introduction

The SharePointSDK module contains the following functions:

image

CRUD Operations for SharePoint List items:

Function Description Compatible SharePoint Version
Add-SPListItem Add an item to a SharePoint list 2010, 2013 and SP Online
Get-SPListFields Get all fields of a SharePoint list 2010, 2013 and SP Online
Get-SPListItem Get all list items of a SharePoint list or a specific item by specifying the List Item ID 2010, 2013 and SP Online
Remove-SPListItem Delete an item from a SharePoint list 2010, 2013 and SP Online
Update-SPListItem Update one or more field values of a SharePoint list item 2010, 2013 and SP Online

The functions listed above are the core functionalities this module provides. it provides simplified ways to manipulate SharePoint list items (Create, Read, Update, Delete).

Miscellaneous Functions

Function Description Compatible SharePoint Version
Import-SPClientSDK Load SharePoint Client Component SDK DLLs 2010, 2013 and SP Online
New-SPCredential Based on the type of SharePoint site (On-Prem vs SP Online), create an appropriate credential object to authenticate to the Sharepoint site. 2010, 2013 and SP Online
Get-SPServerVersion Get SharePoint server version 2010, 2013 and SP Online

These functions are called by other functions in the modules. It is unlikely that runbook authors will need to use them directly.

SharePoint List Attachments Operations

Function Description Compatible SharePoint Version
Add-SPListItemAttachment Add an attachment to a SharePoint list item 2013 and SP Online
Get-SPListItemAttachments Download all attached files from a SharePoint list item 2013 and SP Online
Remove-SPListItemAttachment Delete an attached file (based on file name) from a SharePoint list item 2013 and SP Online

As the names suggest, these functions can be used to manage attachments for SharePoint list items.

I’d like to point out  that the Add-SPListItemAttachment function not only support uploading an existing file to the SharePoint list item. it can also be used to create an attachment file directly using a byte array. This function can be used in 3 scenarios:

  • Uploading an existing file from the file system
  • Directly creating a text based file with some contents as a list item attachment.
  • Read the content of an existing binary (or text)  file, save it as a attachment with a different name

 

Configuration Requirements

Download and Prepare the module

The module zip file should consist the following 5 files:

image

  • Microsoft.SharePoint.Client.dll – One of required DLLs from the SDK
  • Microsoft.SharePoint.Client.Runtime.dll – One of required DLLs from the SDK
  • SharePointSDK.psd1 – Module Manifest file
  • SharePointSDK.psm1 – PowerShell module file
  • SharePointSDK-Automation.json – SMA Integration Module Meta File (where the connection asset is defined).

Download SharePointSDK Module

Note:

The zip file you’ve downloaded from the link above DOES NOT contain the 2 DLL files. I am not sure if Microsoft is OK with me distributing their software / intellectual properties. So, just to cover myself, you will need to download the SDK (64-bit version) from Microsoft directly (https://www.microsoft.com/en-us/download/details.aspx?id=35585), install it on a 64-bit computer, and copy above mentioned 2 DLLs into the SharePointOnline module folder.

Once the SDK is installed, you can find these 2 files in “C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\” folder.

Once the DLLs are placed into the folder, zip the SharePointSDK folder to SharePointSDK.zip file again, and the integration module is ready.

image

Import Module

Once the DLLs are zipped into the module zip file, import the module into SMA by using the Import Module button under Assets tab

image

Create a Connection object to the SharePoint site

After the module has been successfully, a connection to SharePoint Site must be created. The Connection type is “SharePointSDK”

image

The following fields must be filled out:

  • Name: Name of the connection.
  • SharePointSiteURL: URL to your sharepoint site
  • UserName : a User who should be part of the site members role (members group have contribute access).
    • If the site is a SharePoint Onine site, this username MUST be in the email address format. (i.e. yourname@yourcompany.com). I believe this account must be an account created in the Office 365 subscription. I have tried using an outlook.com account (added as a SharePoint site member), it didn’t work.
    • When connecting to a On-Prem SharePoint site, you can use the Domain\UserName format (As shown in the screenshot below)
  • Password: Password for the username you’ve specified.
  • IsSharePointOnlineSite: Boolean field (TRUE or FALSE), specify if it is a SharePoint Online site.

i.e. the connection to a SharePoint site in my lab:

image

Sample Runbooks

In order to better demonstrate this module, I have also created 10 sample runbooks:

image

Download Sample runbooks

I’ll now go through each sample runbook.

Runbook: Sample-SPNewUserRequestList

This sample runbook creates a brand new dummy new users requests list on your SharePoint site. The list created by this runbook will then be used by other sample runbooks (for demonstration purposes).

This runbook is expecting 2 input parameters:

  • ListName: The Display Name that you’d like to name the new users requests list (i.e. New Users OnBoarding Requests).
  • SPConnection: The name of the SharePointSDK connection that you’ve created previously (i.e. Based on the connection I’ve created in my lab as shown previously, it is “RequestsSPSite”

image

This runbook creates a list with the following fields:

image

Runbook: Sample-SPGetListFields

This runbook demonstrates how to retrieve all the fields of a particular list.

image

Runbook: Sample-SPAddListItem

This runbook adds an item to the New Users Requests list the previous runbook created. It also demonstrates how to create a text file attachment directly to the list item (without having the need for an existing file on the file system).

It is expecting the following inputs:

  • Title (New users title, i.e. Mr. Dr. Ms, etc)
  • FirstName (New user’s first name)
  • LastName (New user’s last name)
  • Gender (New user’s Gender: Male / Female)
  • UserName (New user’s user vname)
  • AttachmentFileName (file name of the text based attachment)
  • TextAttachmentContent (content of the text file attachment)
  • NewUserListName (display name of the new users requests list. i.e. New Users OnBoarding Requests)
  • SPConnection (The name of the SharePointSDK connection that you’ve created previously (i.e. Based on the connection I’ve created in my lab as shown previously, it is “RequestsSPSite”)

i.e.

image

The list item is created on SharePoint:

SNAGHTML1d5a6df8

Attachment content:

image

Runbook: Sample-SPUpdateListItem

This runbook can be used to update fields of an existing list item on the New Users Requests list.

Runbook: Sample-SPGetAllListItems

This runbook can be used to retrieve ALL items from a list. Each list item are presented as a hash table.

image

Runbook: Sample-SPGetListItem

This runbook can be used to retrieve a single item from a list.

image

Runbook: Sample-SPDeleteListItem

This runbook deletes a single list item by specifying the List Item ID.

Runbook: Sample-SPAddListItemAttachment

This runbook demonstrates 2 scenarios:

  • Directly attaching a file to a list item
  • attach and rename a file to a list item

image

image

Runbook: Sample-SPDeleteListItemAttachments

This runbook demonstrates how to delete an attachment from a list item (by specifying the file name).

Runbook: Sample-SPDownloadListItemAttachments

This runbook demonstrates how to download all files attached to a list item:

image

Files downloaded to the destination folder:

image

Benefit of Using the SharePointSDK Module

Using as a Regular PowerShell Module

As we all know, SMA modules are simply PowerShell modules (sometimes with optional SMA module meta file .json for creating connections). Although this module is primarily written for SMA, it can also be used in other environments such as a regular PowerShell module or in Azure Automation. When using it as a normal PowerShell module, instead of passing the SMA connection name into the functions inside the module, you may provide each individual value separately (Username, password, SharePoint Site URL, IsSharePointOnlineSite).

Simplified scripts to interact with SharePoint

When using this module, most of the operations around the list item only takes very few lines of code.

i.e. Retrieving a list item:

Using PowerShell:

Using PowerShell Workflow (in SMA):

If you use SharePoint 2013’s REST API, the script will be much longer than what I’ve shown above.

Same Code for Different SharePoint Versions

The SharePoint REST API has been updated in SharePoint 2013. Therefore, if we are to use the REST API, the code for Share Point 2013 would look different than SharePoint 2010. Additionally, when throwing SharePoint Online into the mix, as I mentioned previously, it requires different type of credential for authentication, it further complicates the situation if we are to use the REST API. This makes our scripts and runbooks less generic.

By using this SharePointSDK module, I am able to use the same runbooks on SharePoint 2010, 2013 and SharePoint Online sites.

Limitations

During testing, I noticed the 3 attachments related functions in the SharePointSDK module would not work on SharePoint 2010 sites. These functions are:

  • Add-SPListItemAttachment
  • Remove-SPListItemAttachment
  • Get-SPListItemAttachments

After a bit of research, looks like it is a known issue. I didn’t think it too much a big deal because all the core functions (CRUD operations for the list items) work with SharePoint 2010. Therefore, in these 3 functions, I’ve coded a validation step to exit if the SharePoint Server version is below version 15 (SharePoint 2013):

image

Conclusion

If you are using SMA and SharePoint together, I strongly recommend you to download this module and the sample runbooks and give it a try. If you have a look at the sample runbooks, I’m sure you will realise how easy it is to write PowerShell code interacting with SharePoint.

In case you didn’t see the download links, you can download them here:

Download SharePointSDK Module

Download Sample Runbooks

Lastly, I’m not a SharePoint specialist. If you believe I’ve made any mistakes in my code, or there is room for improvement, I’d like to hear from you. Please feel free to drop me an email Smile.

Using Royal TS for PowerShell Remote Sessions

Written by Tao Yang

Background

I have used many Remote Desktop applications in the past. I have to say Royal TS is the one that I like the most! Recently, I showed it to one of my colleagues, after a bit of playing around, he purchased a license for himself too.

Today, my colleague asked me if I knew that Royal TS is also able to run external commands, and he thought it’s pretty cool that he’s able to launch PowerShell in the Royal TS window. Then I thought, if you can run PowerShell in Royal TS, we should be able to establish PS remote sessions in Royal TS too. Within 10 minutes, we managed to create few connections in Royal TS like these:

SNAGHTML1c209a8d

SNAGHTMLa497d178

image

SNAGHTML1c2e5543

In this post, I’ll go through the steps I took to set them up.

Connections to Individual Servers

To create a connection to an individual server,

01. Choose add->External Application:

image

02. Enter the following Details:

Display Name: The name of the server you want to connect to.

Command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Arguments: -NoExit -Command “Enter-PSSession $CustomField1$”

Working Directory: C:\Windows\System32\WindowsPowerShell\v1.0

On the icon button next to the display name, choose “Use Application Icon” if you want to.

image

image

03. Choose a Credential if you want to connect using an alternative credential

SNAGHTML1c5136c4

If you choose to use an alternative credential,  you must also tick “Use Credentials” box under Advanced tab:

image

04. Enter the remote server name in Custom Field 1:

image

Note: in the arguments field from step 01, I’ve used a Royal TS variable $CustomField1$ as the name of the computer in the Enter-PSSession command. It is more user friendly to use the Custom Field for the computer name, rather than modifying the argument string for each connection that you wish to create.

Create An Ad-Hoc Connection

You can also create a connection in Royal TS for Ad-Hoc connections. In this scenario, you will need to enter the remote computer that you wish to connect to:

image

After the the computer name has been entered, the connection is then established:

image

To create this connection in Royal TS, instead of using the Custom Field 1 for the computer name, I’ve added an additional PowerShell command in the Arguments:

Arguments: -NoExit -Command “$Computer = Read-Host ‘Please enter the Computer Name'; Enter-PSSession $Computer”

image

The Custom Field 1 is no longer required in this scenario. Everything else is the same as the previous sample (for individual computers).

Other Considerations

Maximised PowerShell Window

You may have noticed from the screenshots above, that the PowerShell windows are perfectly fitted in the Royal TS frame. this is because I am also using a customised PS Module that I’ve written in the past to resize the PoewerShell window. Without this module, the PowerShell console would not automatically fit into the Royal TS frame:

image VS image

If you like your console looks like the left one rather than one on the right, please follow the instruction below.

01. Download the PSConsole Module and place it under C:\windows\system32\WindowsPowerShell\v1.0\Modules

image

02. Modify the “All Users Current Host” profile from a normal PowerShell window (NOT within PowerShell ISE). If you are not sure if this profile has been created, run the command below:

image

After the profile is created, open it in notepad (in PowerShell window, type: Notepad $Profile.AllUsersCurrentHost) and add 2 lines of code:

image

After saving the changes, next time when you initiate a connection in Royal TS, the console will automatically maximise to use all the usable space.

Note: Because most likely you will be using an alternative (privileged credential) for these PS remote sessions. therefore the resize console commands cannot be placed into the default profile (current user current host). It must be placed into an All users profile. And also because the resize command only works in a normal PowerShell console (not in PowerShell ISE), therefore the only profile that you can use is the “All Users Current Host” profile from the normal PowerShell console.

Alternatively, if you do not wish to make changes to the All Users Current host profile, you can also add the above mentioned lines into the Royal TS connection arguments field:

i.e.

Arguments: -NoExit -Command “import-module psconsole; resize -max; Enter-PSSession $CustomField1$”

image

Duplicating Royal TS Connections

If you want to create multiple connections, all you need to do is to create the first one manually, and then duplicate it multiple times:

image

When duplicating connections, the only fields you need to change are the Display Name and CustomField1.

WinRM configuration

Needless to say, WinRM must be enabled and properly configured for PS remoting to work. this is a pre-requisite. I won’t go through how to configure WinRM here. Someone actually wrote a whole book on this topic.

Conclusion

I’d like to thank Stefan Koell (blog, twitter), the Royal TS developer (and also my fellow SCCDM MVP) for such an awesome tool. This is now probably THE most used application on all my computers Smile.

If you haven’t tried Royal TS out, please give it a try. Other than the obvious Windows version, there are also a Mac version, an iOS version and an Android version.

A Free Management Pack Catalog for Everyone

Written by Tao Yang

In the 2 most recent post, I have blogged my experience setting up a Microsoft OpsMgr MP catalog using SharePoint and SMA. I managed to produce 2 versions:

  • On-Premise Version: Using SharePoint 2013 and SMA (System Center Service Management Automation)
  • Off-Premise (Cloud) Version: Using Office 365 SharePoint Online and Azure Automation

As I mentioned at the end of the second post, I was working with my SCCDM MVP friend Dan Kregor to create this MP catalog on Sparq Consulting’s public SharePoint site – Free for everyone.

I am pleased to announce, everyone can now access this catalog from this URL:

http://sharepoint.sparqconsulting.com.au/mpcatalog

image

This catalog is publicly available, no login is required. Consider it as a Christmas gift from us Smile.

This catalog is hosted on SharePoint Online, and I have scheduled the Azure Automation runbook to run daily at 9:00pm of my local time to Synchronise with Microsoft’s MP Catalog (It’s Australia Eastern Standard Tiime).

If you like what we’ve done for the System Center community, please help us by spreading the words around, and we will be much appreciated if you want to link to this URL from your websites.

Lastly, other than Dan Kregor, I’d also like to thank all OpsMgr focused MVPs who’s been involved in this discussion since beginning. Thank you for all your input and feedback.

Merry Christmas, everyone.

Using Azure Automation to Build Your Own Microsoft OpsMgr MP Catalog on SharePoint Online

Written by Tao Yang

Background

Previously, I have posted how to build your own Microsoft OpsMgr MP Catalog on SharePoint 2013 using SMA. It is a solution that you can implement on-prem in your own environment if you have existing SharePoint 2013 and SMA infrastructure in place. As I mentioned at the end of the previous post, I have also developed a solution to populate this MP catalog on a Office 365 SharePoint Online site using Azure Automation – a 100% cloud based solution. Because of the differences in APIs between on-prem SharePoint 2013 and SharePoint online, one of the runbooks is completely different than the on-prem version. In this post, I will go through how I’ve setup the MP catalog on SharePoint Online using Azure Automation.

01. Create a List on the SharePoint Online site

The list creation and customization process is exactly the same as the On-Prem SharePoint 2013 version. I won’t go through this again. please refer to Step 1 and the customizing SharePoint List sections in my previous post.

02.  Create a Runbook to Retrieve Microsoft MP info

Again, this runbook is unchanged from the On-Prem version. Simply import it into your Azure Automation account.

Download Get-MSMPCatalog

SNAGHTML803890ed

03. Prepare the SMA Integration Module SharePointOnline

In order to authenticate to SharePoint Online sites, We must use a SharePointOnlineCredentials instance in the script. In my previously post, I wrote a runbook called Populate-OnPremMPCatalog. That runbook utilize Invoke-RestMethod PowerShell cmdlet to interact with SharePoint 2013’s REST API. Unfortunately, we cannot pass a SharePointOnlineCredentials object to this Cmdlet, therefore it cannot be used in this scenario.

Additionally, the SharePointOnlineCredentials class comes from the SharePoint Client Component SDK. In order to create a SharePointOnlineCredentials object in PowerShell scripts, the script need to firstly load the assemblies from 2 DLLs that are part of the SDK. Because I can’t install this SDK in the Azure Automation runbook servers, I needed to figure out a way to be able to load these DLLs in my runbook.

As I have previously written SMA Integration Modules with DLLs embedded in. This time, I figured I can do the same thing – Creating a PowerShell / SMA Integration module that includes the required DLLs. Therefore, I’ve created a customised module in order to load the assemblies. But since the SDK also consists of other goodies, I have written few other functions to perform CRUD (Create, Read, Update, Delete) operations on SharePoint list items. These functions have made the runbook much simpler.

I called this module SharePointOnline, it consists of 5 files:

  • Microsoft.SharePoint.Client.dll – One of required DLLs from the SDK
  • Microsoft.SharePoint.Client.Runtime.dll – One of required DLLs from the SDK
  • SharePointOnline.psd1 – Module Manifest file
  • SharePointOnline.psm1 – PowerShell module file
  • SharePointOnline-Automation.json – SMA Integration Module Meta File (where the connection asset is defined).

image

Download SharePointOnline Module

Note:

The zip file you’ve downloaded from the link above DOES NOT contain the 2 DLL files. I am not sure if Microsoft is OK with 3rd party distributing their software / intellectual properties. So, just to cover myself, you will need to download the SDK (64-bit version) from Microsoft directly (https://www.microsoft.com/en-us/download/details.aspx?id=35585), install it on a 64-bit computer, and copy above mentioned 2 DLLs into the SharePointOnline module folder.

Once the SDK is installed, you can find these 2 files in “C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\” folder.

Once the DLLs are placed into the folder, zip the SharePointOnline folder to SharePointOnline.zip file again, and the integration module is ready.

image

I’d like to also briefly go through this SharePointOnline module. This module contains the following functions:

  • Import-SharePointClientSDK: Load the Assemblies from the 2 DLLs included in the module
  • New-SPOCredential: Create a new SharePointOnlineCredentials object from the username and password provided.
  • Get-SPOListFields: Get all fields from a SharePoint Online list (return an array object)
  • Add-SPOListItem: Add an item to the SharePoint Online list (by passing in a hash table containing the value for each field)
  • Get-SPOListItems: Get all items from a SharePoint Online list (return an array object)
  • Remove-SPOListItem: Remove a list item from a SharePoint Online list (by providing the ID of the item)
  • Update-SPOListItem: Update a list item (by providing the list Item ID and a hash table containing updated values)

This module is made to be re-used for SharePoint Online operations that involves list items. I will write a separate post to go through this in details. But for now, all we need to do is to import it into Azure Automation.

 

04. Import SharePointOnline Module into Azure Automation and Create SharePoint Online Connection

Now that the integration module is ready, it needs to be imported into your Azure Automation account. This is done via the Import Module button under Assets tab.

Once the module is imported, a connection object must also be created.

SNAGHTML806ea00a

image

You must provide the following information when creating the SharePointOnline connection object:

  1. SharePointSiteURL – The URL to your SharePoint Online site (i.e. https//yourcompany.sharepoint.com)
  2. UserName – a User how should be part of the site members role (members group have contribute access). This username MUST be in the email address format. (i.e. yourname@yourcompany.com). I believe this account must be an account created in the Office 365 subscription. I have tried using an outlook.com account (added as a SharePoint site member), it didn’t work.
  3. Password – Password for the username you’ve specified.

i.e.

SNAGHTML80731611

 

05. Create a Runbook to Populate SharePoint List

This is equivalent to the previous runbook Populate-OnPremMPCatalog. I have named it Populate-SPOnlineMPCatalog.

Download Populate-SPOnlineMPCatalog Runbook

SNAGHTML8093a11f

This runbook is expecting 4 parameters:

  • SPOConnection: The name of the SharePointOnline connection that you’ve created earlier.
  • ListName: The list name of your MP catalog list.
  • NotifyByEmail: Specify if you’d like an email notification when new MPs have been added to the catalog.
  • ContactName: If NotifyByEmail is set to “true”, specify the SMAAddressBook connection name for the email notification recipient.

Note: If you’d like to receive email notifications, you also need to import and configure the SendEmail and SendPushNotification modules from my blog. Once the SMTP server connection and the Address book connection are created, please modify line 111 of the runbook with the name of your SMTP server connection:

image

Note: I have previously blogged the issues I have experienced using the SendEmail module in Azure Automation. You may find this post useful: Using the SendEmail SMA Integration Module in Azure Automation.

 

06. Executing Runbook Populate-SPOnlineMPCatalog

When executing the runbook, you need to fill out the parameters listed above:

image

image

Result:

image

Same as the On-Prem version using SMA, you can create a schedule to run this on a regular basis to keep the catalog in sync with Microsoft. I won’t go through the schedule creation again.

Azure Automation Fairshare

Currently, Azure Automation has a “Fairshare” feature, where the maximum allowed execution time for a job is 30 minutes. Fortunately, based on my multiple test runs against multiple Office 365 SharePoint online sites, the first executions of this runbook always complete JUST under 30 minutes. However, if you found your job in Azure Automation is terminated after 30 minutes, you should be able to just run it again to top up the list. But any subsequent runs should only take few minutes.

Conclusion

To me, this post completes the circle. I’m happy that I am able to provide solutions for people who wants to host the catalog On-Premise (by using SharePoint 2013 and SMA), as well as who’d like to hosted in on the cloud (Office 365 and Azure Automation).

The 2 different runbooks (and the additional integration module) are 100% based on what SharePoint are you going to use. There is also a 3rd possible combination: Using SMA to populate SharePoint Online list. In this scenario, the steps are the same as what I described in this post. I have also tested in my lab. it is working as expected.

Additionally, I am also working with my fellow System Center MVP Dan Kregor to make this MP Catalog publicly available for everyone on Sparq Consulting’s public SharePoint Online site. We will make a separate announcement once it is ready. – So even if you can’t setup one up on-prem or on the cloud, we’ve got you covered Smile.

Credit

Thanks for all the System Center MVPs who have provided feedback and input into this solution. Smile

Using SMA to Building Your Own Microsoft OpsMgr Management Pack Catalog On SharePoint 2013

Written by Tao Yang

Background

Over the years that I’ve been working with OpsMgr, for me, the Microsoft Pinpoint MP Catalog was a one stop shop for getting Microsoft management packs. More information about the Pinpoint MP Catalog can be found in Marnix’s post: http://thoughtsonopsmgr.blogspot.com.au/2010/07/pinpoint-management-pack-catalog.html

Based on the information came out in TechEd Europe 2014, looks like there will be changes introduced to the System Center Pinpoint site (http://channel9.msdn.com/Events/TechEd/Europe/2014/Ch9-34, from 09:00 onwards). And it seems the link from Marnix’s post https://pinpoint.microsoft.com/systemcenter is no longer valid.

So I’ve decided to do some experiment, see if I can generate and maintain a Microsoft MP catalog on-premise, for myself – Something I’ve always wanted. It took me couple of days, and I managed to build a MP catalog on a SharePoint 2013 site using PowerShell, SMA, and some existing scripts from the System Center community. The End result looks like this:

SNAGHTML765cc0bc

Currently, this catalog contains 1404 entries, and it is generated by 2 SMA runbooks that I have developed. In this post, I will go through the steps I took to setup this solution.

01. Creating a SharePoint 2013 List

Firstly, I created a list on my SharePoint 2013 site and called it “MP Catalog”:
SNAGHTML7607e74d

image

Once the list is created, go to Site Settings then “Site libraries and lists”

SNAGHTML7609667a

SNAGHTML760b0e10

Choose Customize “MP Catalog” and click on column “Title”

image

Rename this column to “System Name”

image

Add the following additional columns:

  • Categories
  • Catalog Item Id
  • MP Version
  • Public Key
  • Version Independent GUID
  • Download Link
  • Release Date

image

For each of these additional columns, please make sure “Require that this column contains information” is set to “No”.

image

Note: the internal names of these columns would be different than these display names. The script in the SMA runbook will translate these display names to the actual internal names. So it doesn’t matter how you created these columns, as long as the display names are exactly the same as what I listed, it should be OK.

02. Create a Runbook to Retrieve Microsoft OpsMgr Management Packs Info

We have all used the “Download Management Packs” function in the OpsMgr console:

image

Michel Kamp has written a MP that checks updated management packs. This MP utilize the same web service as what “Download Management Packs” wizard uses. I have used some PowerShell code from Michel’s MP in this runbook. –  Thank you, Michel.

I called this runbook Get-MSMPCatalog:

Download Get-MSMPCatalog

SNAGHTML7630c779

This runbook will be called by another runbook, and it returns an arraylist as output.

03. Create a Runbook to Populate the SharePoint List

I created a second runbook to call the first runbook Get-MSMPCatalog, then workout which MPs are not on the SharePoint List, and add the missing ones. I called the second Runbook Populate-OnPremMPCatalog:

Download Populate-OnPremMPCatalog

SNAGHTML763e3f49

As you can see, this runbook is expecting 5 parameters:

  • SharePointSiteURL – URL of the SharePoint site (in my lab, its http://sharepoint01/sites/requests)
  • SavedCredentialName – A credential saved in SMA that has access to the SharePoint site. In my lab, I created an AD service account and assigned it as a member in the SharePoint site.
  • List Name – The list for the MP catalog. In my lab, it’s “MP Catalog”
  • NotifyByEmail – Specify if you’d like an email notification when new MPs have been added to the catalog.
  • ContactName – If NotifyByEmail is set to “true”, specify the SMAAddressBook connection name for the email notification recipient.

Note: If you’d like to receive email notifications, you also need to import and configure the SendEmail and SendPushNotification modules from my blog. Once the SMTP server connection and the Address book connection are created, please modify line 121 of the Populate-OnPremMPCatalog runbook with the name of your SMTP server’s SMA connection:

image

i.e. My SMTP connection and SMAAddress book connection:

SNAGHTML766a1bed

image

04. Execute Runbook Populate-OnPremMPCatalog

When executing this runbook, you will need to fill out some parameters as listed above:

image

image

The first run will take a long time (in my lab, over 1 hour), but any subsequent executions shouldn’t take long at all. i.e. I deleted 204 MPs from the SharePoint list, and execute it again:

image

SNAGHTML76510d6f

I have also created a schedule to execute this runbook daily. This is to make my catalog in sync with Microsoft’s, and notify me when new MPs are released.

image

Customizing the MP Catalog SharePoint List

You may not like the default view of the MP Catalog list. You may want to hide some of the columns. This can be easily done by customising the default view of the list, or creating new views.

image

Conclusion

I’ve always wanted a place where I can simply click on a link to download a particular management pack. I have made this possible by using SMA and SharePoint. The only downside is, only individual management packs are listed. They are not bundled, and no documentations available.

There is also a MP Catalog wiki page on TechNet: https://social.technet.microsoft.com/wiki/contents/articles/16174.microsoft-management-packs.aspx, it is maintained by Microsoft and few System Center MVPs. You should be able to find all the recent MS management packs from there as well.

I have also managed to populate this catalog on a SharePoint Online (Office 365) site using Azure Automation – a 100% cloud based solution. For the cloud based version, one of the runbooks is completely different due to the differences in SharePoint APIs (SharePoint 2013 vs. SharePoint Online). I will post it in few days.

Credit

Thanks for all the System Center MVPs who have provided input and feedback on this topic. You know who you are, much appreciated! Smile

Lastly, please feel free to contact me if you have anything to add on this topic. I’d love to hear from you.

Using the SendEmail SMA Integration Module in Azure Automation

Written by Tao Yang

Over the last couple of days, I’ve spent sometime on Azure Automation (SMA in Azure). The first thing I did was imported and configured the SendEmail and SendPushNotification SMA Integration Modules that I have posted earlier. I created a simple test runbook to send an email and a push notification to my android phone:

However, I found 2 issues related to the SendEmail module. I’ll go through both of the issues in this post.

Issue 1

When I executed this runbook, it failed to send the email message. I got this error:

Cannot find the ‘Send-Email’ command. If this command is defined as a workflow, ensure it is defined before the workflow that calls it. If it is a command intended to run directly within Windows PowerShell (or is not available on this system), place it in an InlineScript: ‘InlineScript { Send-Email }’

SNAGHTML6ef25ba9

I found the cause of this issue is because I did not have a PowerShell module manifest file (psd1) in this module:

image

Whereas the SendPushNotification module works because it has a manifest file:

image

I didn’t pick this one up when I released the modules because it worked in the On-Prem SMA environments when I wrote it. So, it’s easy to fix this issue. I generated a manifest file for SendEmail module, uploaded it to Azure Automation, the issue went away.

Issue 2

After fixing the first issue, I started receiving SMTP authentication errors. I have configured a Gmail account as the sender – same as how I setup in my lab’s SMA environment, but I got SMTP error 5.5.1:

Exception calling “Send” with “1” argument(s): “The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.5.1 Authentication Required.

image

Because this Gmail account is linked to my another Gmail account, I soon received an email from Google telling me they’ve detected some suspicious sign in activities:

SNAGHTML6f0c1c8e

So, looks like a Google security feature has detected someone is trying to sign in not from my normal location (Australia), – because I’ve chosen East US region when I opened my Azure Automation account.

I then decided to use Outlook instead of Gmail. So I created an Outlook account, configured the connection and updated the runbook. Unfortunately, I received similar SMTP errors and the account was temporarily suspended because of these sign in activities.

Luckily, I could go adjust these security activity settings, and verify these sign in activities are mine:

SNAGHTML6f13122c

After adjusting these security settings, the runbook started working and I received the test notification email from the runbook:

SNAGHTML6f13fab7

Conclusion

Based on my experience, I’m guessing the module manifest file is a must-have in Azure Automation? I have updated the SendEmail module and re-uploaded to this blog. If you have already downloaded it, sorry but you will need to download again if you are planning to use it in Azure Automation (Here’s the download link).

And if you are using a public email service as the sender like me, the security features implemented by the service provider may prevent you from using the email account in Azure Automation. You may need to adjust the security settings of the email account (like what I did with the Outlook account).

Lastly, if you haven’t tried Azure Automation, I strongly recommend you to give it a try. You get 500 minutes job run time a month for free (http://azure.microsoft.com/en-us/pricing/details/automation/). This should easily get you started.

VMM 2012 Addendum Management Pack: Detect Failed VMM Jobs

Written by Tao Yang

Background

My MVP friend Flemming Riis needed OpsMgr to alert on failed VMM jobs. After discovering that the native VMM MPs don’t have a workflow for this, I have offered my help and built this addendum MP to alert failed and warning (Completed w/ Info) VMM jobs:

image

I thought it is going to be a quick task, turned out, I started writing this MP about 1 month ago and only able to release it now!

The actual MP is pretty simple, 2 rules sharing a same data source which executes a PowerShell script to detect any failed and warning jobs in VMM. I wrote the initial version in few hours and sent it to Flemming and Steve Beaumont  to test in their environments right before the MVP Summit. After the summit, we found out the MP didn’t work in their clustered VMM environments. We then spent a lot of time emailing back and forth trying to figure out what the issue was. In the end, I had to build a VMM cluster in my lab in order to test and troubleshoot it Smile.

So, BIG BIG “Thank You” to both Flemming and Steve for their time and effort on this MP. It is certainly a team effort!

MP Pre-Requisites

This MP has 2 pre-requisites:

  • PowerShell script execution must be allowed on VMM servers and the VMM PowerShell module must be installed on the VMM server (It should by default).
  • The VMM server must be fully integrated with OpsMgr (configure via VMM console). This integration is required because this integration creates RunAs account to run workflows in native VMM management pack. This Addendum management pack also utilise this RunAs account.

SNAGHTML42d92eab

Alert Rules:

This MP contains 2 alert rules:

  • Virtual Machine Manager Completed w/ Info Job Alert Rule (Disabled by default)
  • Virtual Machine Manager Failed Job Alert Rule (Enabled by default)

image

Both rules shares a same data source with same configuration parameters values (to utilise Cook Down). They are configured to run on a schedule and detects failed / warning jobs since the beginning of the rule execution cycle. i.e. by default, they run every 3 minutes, so they would detect any unsuccessful jobs since 3 minutes ago. An alert is generated for EVERY unsuccessful job:

SNAGHTML42e07b14

SNAGHTML42e1b950

Note: Please keep in mind, If you enable the “Completed w/ Info job alert rule”, because we utilise Cook Down in these 2 rules, if you need to override the data source configuration parameters (IntervalSeconds, SyncTime, TimeoutSeconds), please override BOTH rules and assign same values to them so the script in the data source module only need to run once in every cycle and feed the output to both workflows.

Download

Since it’s a really simple MP, I didn’t bother to write a proper documentation for this, it’s really straight forward, I think I have already provided enough information in this blog post.

Please test and tune it according to your requirements before implementing it in your production environments.

Download Link

Lastly, I’d like to thank Steve and Flemming again for their time and effort on this MP. If you have any questions in regards to this MP, please feel free to send me an email.

My Experience Manipulating MDT Database Using SMA, SCORCH and SharePoint

Written by Tao Yang

Background

At work, there is an implementation team who’s responsible for building Windows 8 tablets in a centralised location (we call it integration centre) then ship these tablets to remote locations around the country. We use SCCM 2012 R2 and MDT 2013 to build these devices using a MDT enabled task sequence in SCCM. The task sequence use MDT locations to apply site specific settings (I’m not a OSD expert, I’m not even going to try to explain exactly what these locations entries do in the task sequence).

SNAGHTML4d5244b

In order to build these tablets for any remote sites, before kicking off the OSD build, the integration centre’s default gateway IP address must be added to the location entry for this specific site, and removed from any other locations.

SNAGHTML4dbe1c0

Because our SCCM people didn’t want to give the implementation team access to MDT Deployment Workbench, my team has been manually updating the MDT locations whenever the implementation team wants to build tablets.

I wasn’t aware of this arrangement until someone in my team went on leave and asked me to take care of this when he’s not around. Soon I got really annoyed because I had to do this few times a day! Therefore I decided to automate this process using SMA, SCORCH and SharePoint so they can update the location themselves without giving them access to MDT.

The high level workflow is shown in the diagram below:

MDT Automation

Design

01. SharePoint List

Firstly, I created a list on one of our SharePoint sites, and this list only contains one item:

SNAGHTML52b1a8c

02. Orchestrator Runbook

I firstly deployed the SharePoint integration pack to the Orchestrator management servers and all the runbook servers. Then I setup a connection to the SharePoint site using a service account

SNAGHTML532bed6

The runbook only has 2 activities:

image

Monitor List Items:

SNAGHTML53742a6

Link:

SNAGHTML536dbb9

The link filters the list ID. ID must equal to 1 (first item in the list). This is to prevent users adding additional item to the list. They must always edit the first (and only) item on the list.

Start SMA Runbook called “Update-MDTLocation”:

image

image

This activity runs a simple PowerShell script to start the SMA runbook. The SMA connection details (user name, password, SMA web service server and web service endpoint) are all saved in Orchestrator as variables.

SNAGHTML53aaec0

03. SMA Runbook

SNAGHTML5413f3c

Firstly, I created few variables, credentials and connections to be used in the runbook:

Connections:

Credential:

  • Windows Credential that has access to the MDT database (we have MDT DB located on the SCCM SQL server, so it only accepts Windows authentication). I named the credential “ProdMDTDB”

Variables:

  • MDT Database SQL Server address. I named it “CM12SQLServer”
  • Gateway IP address. I named it “GatewayIP”

 

Here’s the code for the SMA runbook:

Putting Everything Together

As demonstrated in the diagram in the beginning of this post, here’s how the whole workflow works:

  1. User login to the SharePoint site and update the only item in the list. He / She enters  the new location in the “New Gateway IP Location” field.
  2. The Orchestrator runbook checks updated items in this SharePoint list every 15 seconds.
  3. if the Orchestrator runbook detects the first (and only) item has been updated, it takes the new location value, start the SMA runbook and pass the new value to the SMA runbook.
  4. SMA runbook runs a PowerShell script to update the gateway location directly from the MDT database.
  5. SMA runbook sends email to a nominated email address when the MDT database is updated.

The email looks like this:

SNAGHTML197a0f95

The Orchestrator runbook and the SMA runbook execution history can also be viewed in Orchestrator and WAP admin portal:

image

image

Room for Improvement

I created this automation process in a quick and easy way to get them off my back. I know in this process, there are a lot of areas can be improved. i.e.

  • Using a SMA runbook to monitor SharePoint list direct so Orchestrator is no longer required (i.e. using the script from this article. – Credit to Christian Booth and Ryan Andorfer).
  • User input validation
  • Look up AD to retrieve user’s email address instead of hardcoding it in a variable.

Maybe in the future when I have spare time, I’ll go back and make it better , but for now, the implementers are happy, my team mates are happier because it is one less thing off our plate Smile.

Conclusion

I hope you find my experience in this piece of work useful. I am still very new in SMA (and I know nothing about MDT). So, if you have any suggestions or critics, please feel free to drop me an email.

Installing VMM 2012 R2 Cluster in My Lab

Written by Tao Yang

I needed to build a 2-node VMM 2012 R2 cluster in my lab in order to test an OpsMgr management pack that I’m working on. I was having difficulties getting it installed on a cluster based on 2 Hyper-V guest VMs, and I couldn’t find a real step-to-step detailed dummy guide. So after many failed attempts and finally got it installed, I’ll document the steps I took in this post, in case I need to do it again in the future.

AD Computer accounts:

I pre-staged 4 computer accounts in the existing OU where my existing VMM infrastructure is located:

  • VMM01 – VMM cluster node #1
  • VMM02 – VMM cluster node #2
  • VMMCL01 – VMM cluster
  • HAVMM – Cluster Resource for VMM cluster

SNAGHTML14878767

I assign VMMCL01 full control permission to the HAVMM (Cluster resource) computer AD account:

SNAGHTML148c6725

IP Addresses:

I allocated 4 IP addresses, one for each computer account listed above:

image

Guest VMs for Cluster Nodes

I created 2 identical VMs (VMM01 and VMM02) located in the same VLAN. There is no requirement for shared storage between these cluster nodes.

Cluster Creation

I installed failover cluster role on both VMs and created a cluster.

image

image

image

image

image

VMM 2012 R2 Installation

When installing VMM management server on a cluster node, the installation will prompt if you want to install a highly available VMM instance, select yes when prompted. Also, the SQL server hosting the VMM database must be a standalone SQL server or a SQL cluster, the SQL server cannot be installed on one of the VMM cluster node.

DB Configuration

image

Cluster Configuration

image

DKM Configuration

image

Port configuration (left as default)

image

Library configuration (need to configure manually later)

image

Completion

image

Run VMM install again on the second cluster node.

As instructed in the completion window, run ConfigureSCPTool.exe –AddNode HAVMM.corp.tyang.org CORP\HAVMM$

Cluster Role is now created and can be started:

image

OpsMgr components

In order to integrate VMM and OpsMgr, OpsMgr agent and console need to be installed on both VMM cluster node. I pointed the OpsMgr agent to my existing management group in the lab, approved manually installed agent and enabled agent proxy for both node (required for monitoring clusters).

Installing Update Rollup

After OpsMgr components are installed, I then installed the following updates from the latest System Center 2012 R2 Update Rollup (UR 4 at the time of writing):

  • OpsMgr agent update
  • OpsMgr console update
  • VMM management server update
  • VMM console update

Connect VMM to OpsMgr

I configured OpsMgr connection in VMM console:

2014-11-19_22-29-43

 

Conclusion

The intention of this post is simply to dump all the screenshots that I’ve taken during the install, and document the “correct” way to install VMM cluster that worked in my lab after so many failed attempts.

The biggest hold up for me was without realising I need to create a separate computer account and allocate a separate IP address for the cluster role (HAVMM). I was using the cluster name (VMMCL01) and its IP address in the cluster configuration screen and the installation failed:

image

After going to through the install log, I realised I couldn’t use the existing cluster name:

image

When I ran the install again using different name and IP address for the cluster role, the installation completed successfully.