Tag Archives: MimboloveOrchestrator
Over the last week or so, I’ve been busy creating Squared Up dashboards for various System Center 2012 components, which will be made publicly available through Squared Up’s new community site. While I was creating dashboards for some System Center components, I realised how little do the native MP from Microsoft offers (i.e. Orchestrator and Windows Azure Pack). Luckily, some awesome MP developers have recognised this and released additional MPs to fill the gaps. i.e
- Brian Wren’s (aka MP Author) Orchestrator Runbook Sample MP: https://gallery.technet.microsoft.com/Orchestrator-Runbook-90307b26
- Oskar Landman’s Windows Azure Pack (WAP) MP: https://gallery.technet.microsoft.com/SCOM-Management-Pack-3855607d
So in additional to creating Squared Up dashboards for the native System Center management packs, I have also created dashboards for above mentioned community MPs.
However, if you have used (or tried to use) Brian’s Orchestrator Runbook MP, you’d probably know the PowerShell scripts in the MP is not compatible with Windows Server 2012 R2 or PowerShell version 3 and later. The issue is raised in the Q and A section (in a function, you can not use return in try-catch-finally statement).
I have tried to implement this MP a while back at my previous job. Not only I tried to update the PowerShell scripts, in my opinion, I did not want to create another “Runbook Host” class (by creating few registry keys and values) for hosting runbooks. Therefore, I’ve made another update to this MP: I’ve removed “Runbbook Host” class, and configured the “Orchestrator Runbooks” objects to be hosted by the Orchestrator Management Server:
You can also see the hosting stack in Squared Up (the critical object on the top is the runbook, and the critical object at the bottom is the Windows Computer:
After I updated this MP (must be over a year ago), I’ve never bothered to publish it. Few weeks ago, my fellow SCCDM MVP Adin Ermie had a requirement for a fixed version. That’s when I realised I worked on it last year. When I gave Adin my version, he has also found out I fixed some scripts, but not all of them. So I updated it again, tested it myself today, and made sure the runbooks are correctly discovered and monitored.
If you have already imported the original MP in your management group, I am afraid my updated version will not be compatible for in place upgrade, because I have removed various elements from the original MP (class definition, discoveries, etc.), and I have also used another key to seal the MP. These changes made the updated MP not compatible for in-place upgrade. So, if you’d like to use my version, you will have to delete the original version from your management group first.
Configuring the Management Pack
From Brian’s original MP download, there is also a MP guide included. As stated in the original MP guide, you will need to create few registry keys and values for the Runbook Host as well as creating a Run As account that has access to the Orchestrator Web Service.
When using this updated MP, it is obvious that you do not need to create those registry keys and values anymore because the “Runbook Host” class has been removed. But you will still need to create this Run As account and assign it to the “Orchestrator Web Service Account” Run As Profile:
However, since I have removed the “Runbook Host” class, instead of distributing this Run As account to the health service hosting the “Runbook Host” class instance, you will need to distribute it to your Orchestrator management servers instead.
You may also want to monitor the “Monitor runbooks”. If this is the case, you will need to enable the “Runbook Running” monitor for the “MPAuthor Monitor Runbooks” group:
I have enabled this monitor in my lab, and the Squared Up dashboard I created ended up look like this:
Obviously, big thank-you would go to Brian Wren, for developing this MP and made the entire VSAE project available for everyone. I’d also like to thank my fellow SCCDM MVP Adin Ermie for testing it for me.
You can download my updated MP from the link below. Please feel free to contact me if you have any questions or issues.
Back in August / September last year, I spent sometime designed a set of Orchestrator runbooks to migrate Window Server 2008 R2 Hyper-V clusters to Windows Server 2012. I wasn’t going to blog this because it was designed to only cater for my company’s environment, not something that’s generic enough for everyone to use. I also wasn’t sure how well I can explain and document these runbooks in a blog post. Few of my colleagues and friends actually encouraged me to blog this one so I’ll give it a try (and not to disclose company related sensitive information).
My employer is one of the biggest two supermarkets and retail chains in Australia. Few years ago, there was a project that implemented a 2-node Windows Server 2008 R2 Hyper-V cluster in each of the 750 supermarkets that owned by my employer. More details about the implementation can be found from this Microsoft Case Study.
Early last year, we have started a project to upgrade the entire System Center suite to System Center 2012 (later on decided to go to R2). A part of this project is also to upgrade 2008 R2 Hyper-V clusters in supermarkets to Windows Server 2012 (and later on decided to go straight to Windows Server 2012 R2).
I have been in my current role for just over 2.5 years now, one thing that I learnt while managing a large retail environment is, that Automation is THE key to our success. No matter what solutions you are planning to rollout to the stores, if you can’t automate the rollout process, your plan is not going to fly. Therefore scripting is pretty much the essential requirement for everyone in my team. And in the office, you hear the phrase “cookie cutter” a lot. Automation standardise and simplifies implementation processes for new technologies. Our job is pretty much to design all kinds of cookie cutters and hand them over to other teams to cut cookies.
For the Hyper-V cluster upgrade, our end goal is that the implementers would enter a store number in the System Center Service Manager 2012 web portal and Service Manager would then kick off a series of Orchestrator runbooks to perform the upgrade. The upgrade would involve the following steps:
- Live migrates all VMs to cluster node 2.
- Evict Node 1 from the 2008 R2 Cluster and rebuild it to Windows Server 2012 R2 using ConfigMgr OSD.
- Migrate all cluster resources from the old cluster on Node 2 (Running Windows 2008 R2) to the new cluster on Node 1.
- Rebuild Node 2 to Windows Server 2012 R2
- Join Node 2 the the new cluster.
I have been asked to assist to design a prototype for step 3 using Orchestrator 2012.
Note: At the time designing this prototype, Windows Server 2012 R2 and System Center 2012 R2 was yet to be released and we had not made decisions to go straight to R2. Therefore my prototype was based on the scenario that we were going to migrate from Windows Server 2008 R2 to 2012 RTM. I had to make some modifications once we have decided to go to 2012 R2. I’ll explain it later in this article.
I firstly scripted the entire cluster migration process in PowerShell, using WinRM and Windows Server 2012 Server Migration Tool. Once the PowerShell script was working 100%, I then broke it up into many small segments and build the Orchestrator runbooks based on this script.
The purpose of this blog post to document my experience designing this prototype. I will provide the runbooks export at the end of this article.
In my prototype, there are 6 runbooks in total. The migration process would go from 01—>02—>03—>04—>05—>06:
01. Data Gathering
02. Prerequisites Checks
03. Export HyperV Config
04. Move Cluster Resources
05. Import HyperV Config
06. Add VMs to the Destination Cluster
These runbooks require the following prerequisites:
- The Orchestrator Runbook Service account needs to have local admin rights on both Hyper-V cluster nodes (this is because it’s only a prototype so I didn’t bother to setup proper service accounts for each activity).
- Hyper-V PowerShell module is installed on both old and new cluster nodes (Microsoft did not provide a Hyper-V PS module out of the box in Windows Server 2008 R2. So for 2008 R2, we used the one from CodePlex).
- WinRM is enabled on both cluster nodes.
01. Data Gathering
The purpose of the first runbook “Data Gathering” is to connect to both cluster nodes and gather few information for the subsequent runbooks. The only information that the operator needs to provide is the server names for both nodes.
It doesn’t matter which node is node 1 and which one is node 2, the runbooks will figure out the source and destination clusters by comparing the OS versions.
The very first step for runbook 01 is to check if the account it’s running under has local admin rights on both clusters. It will only continue of if it can connect to the admin$ share on both clusters.
This runbook will also get the FQDN, OS versions, WinRM ports from each cluster. With regards to WinRM port, because VMM 2008 and VMM 2012 use different WinRM ports when installing VMM agents on the Hyper-V hosts, so in order for other runbooks to connect to each Hyper-V cluster, we need to know which port to connect to (in 2008, port 80 is used and in 2012, it’s the default port of 5985).
02. Prerequisite Checks
This runbook performs the following checks:
01. OS Version Validation – Both cluster nodes cannot be on the same version, and the minimum OS major version must be 6.1 (Windows Server 2008 R2).
02. Cluster Validation – Make sure Windows Failover cluster role is installed on both nodes.
03. Identify Source & Destination Cluster – based on the Windows OS major version, the one with lower version is the source cluster, and the higher version one is the destination cluster.
04. HyperV Validation – Make sure Hyper-V role is installed on both cluster nodes.
05. Check Cluster Nodes Count – Make sure both source and destination clusters only have 1 node at this time.
06. Smig (Server Migration Tool) Feature Validation – Checks if Server Migration Tool is enabled on both clusters and if it is enabled, are both clusters running on the same version of Server Migration tool.
If step 6 (Smig Feature Validation) failed, Step 7 to Step 10 are performed to configure Smig on both clusters:
07. Create Smig Package on Destination Cluster – this step uses “smigdeploy.exe” on the destination cluster to create a smig package based on source cluster’s OS version.
08. Copy Smig Package To Source Cluster – This step copies the package on the destination cluster created in step 7 to the source cluster
09. Register Smig On Source Cluster – After step 8 has copied the package to the source cluster, this step registers the smig package on the source cluster.
10. Smig Feature Re-Validation – performs the same validation as step 6. by now, the server migration tool should be fully configured on both source and destination clusters.
11. Process Prerequisites Checks Result – consolidates results from each prerequisite checks. Continue to Runbook 03 if passed all prerequisite checks.
03. Export HyperV Config
This runbook exports the HyperV configurations on the source cluster using Server Migration tool and then copies the export to the source cluster. The following activities are performed in this runbook:
01. Shutdown VMs on source cluster – This activity shuts down all virtual machines that are currently running on the source Hyper-V cluster.
02. Remove Cluster Resources on Source cluster – because Server Migration tool does not support migrating clusters, so all virtual machine cluster resources needs to be removed from the source cluster.
03. Export HyperV Config From Source Cluster – now, none of the virtual machines are hosted on cluster, they are rather standalone VM’s (Not HA’d). we can now export Hyper-V configurations on the source cluster node using the Server Migration tool Powershell cmdlet Export-SmigServerSetting.
04. Delete Previous Export From Destination Cluster – if there is a copy of previously created Hyper-V smig export on the destination cluster, this step will delete it so we can then copy the most recent copy to destination cluster node next.
05. Copy Hyper-V Export to Destination Cluster – copy the export created in step 3 to the destination cluster node.
Note: for all file copy activities in my runbooks, I couldn’t run them within a Powershell remote session because I am not using CredSSP in my runbooks so I can’t connect to a remote UNC path within a PS remote session – because of the second-hop constraint. Therefore I’m simply running the robocopy command:
I remember I read somewhere that Orchestrator uses PsExec.exe under the hood when running a command against a remote computer. PSExec.exe uses RPC rather than WinRM.
06.Verify HyperV Export Copy on Destination Cluster – after the Hyper-V export has been copied to the destination cluster, this runbook verifies the export files by comparing the file count between the copies located on the source and destination locations. if this validation is passed, the next runbook is then triggered.
04. Move Cluster Resources
This runbook moves cluster resources (cluster shared volumes) to the destination cluster.
01. Delete VM’s on Source Cluster – This step deletes all virtual machines from the source node. at this point, the VMs are not hosted on the source cluster anymore, they are standalone non-HA’d VMs. they need to be deleted from Hyper-V now.
02. Move CSV’s – This steps moves all cluster shared volumes from the source cluster to the destination cluster. this is a tricky one. In order to keep track of the CSV names, I had to move only one CSV at a time. To move it, I firstly save the name of the CSV to a variable, then delete it from the source. after it’s deleted, I can then pick up this particular CSV as available storage in the new cluster. After I added the available storage in the new cluster, I rename the CSV to the name I captured from the source cluster. This is why we can only move one CSV at a time, if we move more than one, there is no way I can identify which one is which in the new cluster. For more information about this process, please refer to the PS script in this activity.
03. Delete CSV’s From Source Cluster – Now that the CSV’s are all moved to the destination, they can be deleted from the source cluster.
04. Delete Available Storage From Source Cluster – After the CSVs are deleted from the source, they are still showing as available storage in the source cluster. This is VERY IMPORTANT, these available storage MUST BE deleted. Without this step, Server Migration Tool will not work 100%. I got stuck here for few days when I was designing the runbooks. Server migration tool does not like clusters. without this step, I was keep getting NTFS related errors when importing the Hyper-V export to the destination cluster using Server migration tool. but if I ran the import again after the first attempt failed, it would work second time. After I added this step in the runbook, the problem went away. so in another word, everything has to be completely removed from the source cluster before we can import the Hyper-V configuration to the destination cluster.
05. Import HyperV Config
now that we have all the darts lined up, it’s time to import the Hyper-V configuration to the destination cluster. This runbook only contains one activity:
01. Import HyperV Config on Destination Cluster – the script in this step firstly make sure all CSV’s are online, then perform the import using Import-SmigServerSetting cmdlet from Server Migration Tool PS snap-in. The HyperV export created from the source cluster also contains information such as Hyper-V virtual switches. after I imported the config to the destination cluster, I do not have make any additional configuration changes as Server Migration tool took care of everything.
06. Add VMs to the Destination Cluster
Now that all VMs are migrated to the destination cluster node. this runbook add them to the destination cluster and starts them up. This runbook contains 2 steps:
01. Add VMs to Cluster – This step adds all VMs to the cluster.
02. Start VMs – This steps powers on all VMs.
Test Drive Results
when we tested this prototype in our lab, with 3-4 VMs hosted on the cluster, These runbooks took around 10-15 minutes to run. – And all the wait I hardcoded in my scripts contributed to this time frame as well. It is completely automated, all we did was to enter the NetBIOS names of the 2 cluster nodes to kick off the first runbook, and sat back, watching Orchestrator to progress with each activity.
Story about Windows 2012 R2
So I finished the prototype by the end of September last year and went on holidays for 4 weeks. Windows 2012 R2 was released while I was on holidays. After I came back, my colleague told me these runbooks didn’t work when migrating from Windows Server 2008 R2 to 2012 R2. because Server Migration Tool only supports migrating from n-1 version when come to Hyper-V. When creating the 2012 R2 version of the smig package for 2008 R2, Hyper-V is not available. However, migration from Windows Server 2012 to 2012 R2 still works using this runbook prototype (because it’s only 1 version behind).
Microsoft advised there is an unsupported undocumented way to make Hyper-V available in the package but we have decided not to use this method because it’s not supported.
After logged a support call with Microsoft and they have advised to just import the Hyper-V VM’s (the xml) to the new cluster. So I spent another couple of hours to modify the existing runbooks to ditch Server Migration Tool and directly import the VM’s XML into Hyper-V. This has made the runbooks simpler than before, but the drawback is that other Hyper-V settings such as virtual switches are not migrated across. Luckily since we have cookie cutters for everything, it’s a no brainer for us to make sure the Hyper-V host servers base build includes the virtual switches creation.
What happened next?
I handed the prototype to some other people and they have then spent few months enhancing it and built the rest of the store upgrade / conversion process based on this prototype. I can’t really disclose too many details about this (at least not at this stage). It has now grown to a monster with many runbooks and integrated with few other System Center 2012 products.
The purpose of this post is to share my experience of designing this specific “cookie cutter” prototype. I hope this would help someone when performing similar kind of tasks.
Both versions of the runbooks can be downloaded from the links below:
Note: I have removed all the email activities from the runbooks exports from the links above.
Just another quick note about the runbooks – When I created these runbooks, I created a folder called “HYPER-WEE”, as a joke. After all the runbooks were finalised, I realised it’s too hard to rename this folder because it’s hardcoded in each Invoke Runbook activity:
I didn’t bother to rename the folder and update all the runbooks. So if you have downloaded and imported the runbook, you’ll see it:
Lastly, as always, please feel free to contact me for any questions or issues.
I was thinking about how I can make my study room quieter and cooler as I had 4 desktop machines under my desks. The VMM server in my lab was running on a very old PC. Couple of weeks ago I bought a HP Proliant N54L Microserver and rebuilt the VMM server on it. I put a 128GB SSD (For OS and all the apps), a 1TB SATA (For VMM Library) and 2x4GB DDR3 memory sticks on it, it runs so quiet, I can hardly hear it.
HP Proliant N54L:
Yesterday, I was going through the labs for Microsoft exam 70-246: Minotoring and Operating a Private Cloud with System Center 2012 and I ran into issues that I could not create a VMM connector to connect to my new VMM server from Service Manager. I got a very simple error message: “Cannot connect to VMM server <vmm server name>”
While I was troubleshooting it, I realised 2 other issues:
1. I couldn’t create a PowerShell remote session to my VMM server using PowerShell command like:
New-PSSession –ComputerName <VMM server FQDN> –Credential <Service Manager VMM Connector Account>
2. After I have created a connection to the new VMM server in the Orchestrator VMM integration pack, I modified an existing test runbook called “Get Cloud” to use the new connection, and it also failed.
All of these worked fine on the old VMM server, WinRM is configured exactly the same between old and new VMM severs because it configured in a domain GPO.
Both these errors were related to some kind of Kerberos authentication errors. (It was very late at night, I forgot to take screenshots while I was troubleshooting it).
After few hours troubleshooting, I have found the problems:
1. During VMM install, I installed SQL server on the VMM server. I somehow installed SSRS (SQL Server Reporting Services) as well although it’s not required for VMM. I had SSRS running on a service account and I have registered the SPN’s for SSRS (i.e. setspn.exe –A http/<VMM server Name> <Service Account>). I realised SSRS was installed by mistake, so I uninstalled it. As soon as I removed the SPN’s for the http service (for SSRS), PowerShell remote sessions started to work. – because PS Remoting also uses http.
2. I then tried to run the VMM PowerShell cmdlet Get-SCVMMServer in the remote shell against the VMM server and I got an error:
The type or name syntax of the registry key value IndigoTcpPort under Software\Microsoft\Microsoft System Center Virtual Machine Manager Administrator Console\Settings is incorrect.
Luckily I found this blog article. In my VMM server’s registry, the value IndigoTcpPort under HKLM\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Administrator Console\Settings is a REG_SZ (string) with no value configured. After I deleted it and recreated a REG_DWORD with value 1fa4 (which is hexadecimal for 8100), everything started working. The Get-SCVMMServer cmdlet worked fine. Service Manager VMM Connector was successfully created and Orchestrator runbooks were able to run.
So to summarise what I’ve done. I’ve reviewed and removed the SPN’s for http service which were originally created for SSRS, and corrected the “IndigoTcpPort” registry value.
It was not how I would like to spend my Saturday night, but I’m glad I’ve got it fixed so I can continue with my study.