Category Archives: SCVMM
My MVP friend Flemming Riis needed OpsMgr to alert on failed VMM jobs. After discovering that the native VMM MPs don’t have a workflow for this, I have offered my help and built this addendum MP to alert failed and warning (Completed w/ Info) VMM jobs:
I thought it is going to be a quick task, turned out, I started writing this MP about 1 month ago and only able to release it now!
The actual MP is pretty simple, 2 rules sharing a same data source which executes a PowerShell script to detect any failed and warning jobs in VMM. I wrote the initial version in few hours and sent it to Flemming and Steve Beaumont to test in their environments right before the MVP Summit. After the summit, we found out the MP didn’t work in their clustered VMM environments. We then spent a lot of time emailing back and forth trying to figure out what the issue was. In the end, I had to build a VMM cluster in my lab in order to test and troubleshoot it .
So, BIG BIG “Thank You” to both Flemming and Steve for their time and effort on this MP. It is certainly a team effort!
This MP has 2 pre-requisites:
- PowerShell script execution must be allowed on VMM servers and the VMM PowerShell module must be installed on the VMM server (It should by default).
- The VMM server must be fully integrated with OpsMgr (configure via VMM console). This integration is required because this integration creates RunAs account to run workflows in native VMM management pack. This Addendum management pack also utilise this RunAs account.
This MP contains 2 alert rules:
- Virtual Machine Manager Completed w/ Info Job Alert Rule (Disabled by default)
- Virtual Machine Manager Failed Job Alert Rule (Enabled by default)
Both rules shares a same data source with same configuration parameters values (to utilise Cook Down). They are configured to run on a schedule and detects failed / warning jobs since the beginning of the rule execution cycle. i.e. by default, they run every 3 minutes, so they would detect any unsuccessful jobs since 3 minutes ago. An alert is generated for EVERY unsuccessful job:
Note: Please keep in mind, If you enable the “Completed w/ Info job alert rule”, because we utilise Cook Down in these 2 rules, if you need to override the data source configuration parameters (IntervalSeconds, SyncTime, TimeoutSeconds), please override BOTH rules and assign same values to them so the script in the data source module only need to run once in every cycle and feed the output to both workflows.
Since it’s a really simple MP, I didn’t bother to write a proper documentation for this, it’s really straight forward, I think I have already provided enough information in this blog post.
Please test and tune it according to your requirements before implementing it in your production environments.
Lastly, I’d like to thank Steve and Flemming again for their time and effort on this MP. If you have any questions in regards to this MP, please feel free to send me an email.
I needed to build a 2-node VMM 2012 R2 cluster in my lab in order to test an OpsMgr management pack that I’m working on. I was having difficulties getting it installed on a cluster based on 2 Hyper-V guest VMs, and I couldn’t find a real step-to-step detailed dummy guide. So after many failed attempts and finally got it installed, I’ll document the steps I took in this post, in case I need to do it again in the future.
AD Computer accounts:
I pre-staged 4 computer accounts in the existing OU where my existing VMM infrastructure is located:
- VMM01 – VMM cluster node #1
- VMM02 – VMM cluster node #2
- VMMCL01 – VMM cluster
- HAVMM – Cluster Resource for VMM cluster
I assign VMMCL01 full control permission to the HAVMM (Cluster resource) computer AD account:
I allocated 4 IP addresses, one for each computer account listed above:
Guest VMs for Cluster Nodes
I created 2 identical VMs (VMM01 and VMM02) located in the same VLAN. There is no requirement for shared storage between these cluster nodes.
I installed failover cluster role on both VMs and created a cluster.
VMM 2012 R2 Installation
When installing VMM management server on a cluster node, the installation will prompt if you want to install a highly available VMM instance, select yes when prompted. Also, the SQL server hosting the VMM database must be a standalone SQL server or a SQL cluster, the SQL server cannot be installed on one of the VMM cluster node.
Port configuration (left as default)
Library configuration (need to configure manually later)
Run VMM install again on the second cluster node.
As instructed in the completion window, run ConfigureSCPTool.exe –AddNode HAVMM.corp.tyang.org CORP\HAVMM$
Cluster Role is now created and can be started:
In order to integrate VMM and OpsMgr, OpsMgr agent and console need to be installed on both VMM cluster node. I pointed the OpsMgr agent to my existing management group in the lab, approved manually installed agent and enabled agent proxy for both node (required for monitoring clusters).
Installing Update Rollup
After OpsMgr components are installed, I then installed the following updates from the latest System Center 2012 R2 Update Rollup (UR 4 at the time of writing):
- OpsMgr agent update
- OpsMgr console update
- VMM management server update
- VMM console update
Connect VMM to OpsMgr
I configured OpsMgr connection in VMM console:
The intention of this post is simply to dump all the screenshots that I’ve taken during the install, and document the “correct” way to install VMM cluster that worked in my lab after so many failed attempts.
The biggest hold up for me was without realising I need to create a separate computer account and allocate a separate IP address for the cluster role (HAVMM). I was using the cluster name (VMMCL01) and its IP address in the cluster configuration screen and the installation failed:
After going to through the install log, I realised I couldn’t use the existing cluster name:
When I ran the install again using different name and IP address for the cluster role, the installation completed successfully.
I was planning to continue on my 70-247 exam preparation tonight, but I couldn’t launch VMM 2012 console from any computers in my lab. So things didn’t turn out as planned. Lucky I’ve figured out the issue now after 4 hours troubleshooting. It’s just passed midnight, still one hour until my bed time, so I thought I’ll quickly document the issue.
When I tried to launch VMM using an account with Full Administrator rights, the console got stuck on below screen during load and would not go any further:
I also found out if I use an account which is a member of a Delegated Admin role that I’ve created called “Cloud Admin”, although it also got stuck on this screen, but the console would eventually load after 10-15 minutes.
Long story short, after spent my entire night troubleshooting, I noticed there were few job got stuck on “Running” state by querying the database using below query:
SELECT * FROM [VirtualManagerDB].[dbo].[tbl_TR_TaskTrail] where TaskState = 'Running'
All of these running jobs were trying to run the VMM cmdlet Get-SCOpsMgrConnection. It seems each of this job is associated to a console connection attempt. I then tried to manually run this cmdlet from a PowerShell console, it also got stuck.
My OpsMgr management group consists 4 management servers and I’ve created a NLB cluster for the Data Access Service. I have configured OpsMgr integration in VMM using the NLB name a while back and the VMM console was working this morning before I went to work. I checked my OpsMgr servers, they are all healthy and I am able to connect to OpsMgr console using the NLB name from a Windows 8 machine. I’m not sure at this stage what’s causing it.
In order to get VMM consoles fixed as soon as possible, I’ve taken the following steps:
1. Kill all VMM console connections via task manager on all the machines that are trying to connect.
2. Restart System Center Virtual Machine Manager service on the VMM server in order to get rid of these running jobs
3. Use PowerShell cmdlet Remove-SCOpsMgrConnetion –force to remove the OpsMgr connection
4. Confirm the OpsMgr connection is removed from the database by running below SQL query:
SELECT * FROM tbl_MOM_OMConnection
5. I restarted VMM service again just to be safe
Now I can open VMM console from any computers on my network
I’ll try to add OpsMgr connection back in later. It’s bed time now…
In the last few days, I’ve been going through the labs from Microsoft’s course 10750A: Monitoring and Operating a Private Cloud with System Center 2012. As part of the lab, I installed the System Center 2012 SP1 version of the Cloud Service Process Pack (CSPP).
Last night, I created a Cloud Resources Subscription Request, and then tried to assign the cloud resources to this request. While I was going through the “Assign Cloud Resources” wizard, I got stuck because my VMM server does not show up in the drop down list:
This has really annoyed me as it’s hard for me to find time to study and prepare for the exam with all the stuff going on at work. The only time I can study is from 9:30pm to 1:00am – after my daughter went to bed… the last thing I want to see at 12:00 mid night is an error stop me from going through the lab.
Anyways, after spending some time on google, I found people are having same issues as me and posted the question on the TechNet forum. As suggested in the forum thread, the issue is the VMM 2012 Discovery management pack.
In my case, this MP was not selected in the SCOM CI connector in Service Manager. I couldn’t select it because it was greyed out in the connector. It was greyed out because the version of the MP loaded in Service Manager was 3.0.6005.0 and the version loaded in OpsMgr is 3.1.6011.0.
According to this article, 3.0.6005.0 is the build number for VMM 2012 RTM and 3.1.6011.0 is the build number for VMM 2012 SP1 RTM. Looks like the VMM MP author is trying to match the VMM build version and MP version.
Version 3.0.6005.0 comes with the SP1 version of the Cloud Service Process Pack. Thus this was the one I imported in to Service Manager.
Version 3.1.6011.0 comes with VMM 2012 SP1 installation (located on the VMM server, under <VMM Installation directory>\ManagementPacks directory), and it has been previously imported into my OpsMgr management group.
After I imported version 3.1.6011.0 to Service Manager (because it’s the latest version and it’s the version in OpsMgr), I was able to select VMM 2012 Discovery MP in the OpsMgr CI connector:
I went to bed after I manually kicked off the connector synchronisation, this morning, the VMM server address appeared in the wizard:
Well, I guess it’s all part of the learning. Since I’m still pretty new to Service Manager, I’m not sure how many people in System Center community already know this issue. This post is more like a note to myself. I’m not sure why Microsoft included a very old version of the VMM MP in the most recent release of the Cloud Service Process Pack (which was only released last month).
I was thinking about how I can make my study room quieter and cooler as I had 4 desktop machines under my desks. The VMM server in my lab was running on a very old PC. Couple of weeks ago I bought a HP Proliant N54L Microserver and rebuilt the VMM server on it. I put a 128GB SSD (For OS and all the apps), a 1TB SATA (For VMM Library) and 2x4GB DDR3 memory sticks on it, it runs so quiet, I can hardly hear it.
HP Proliant N54L:
Yesterday, I was going through the labs for Microsoft exam 70-246: Minotoring and Operating a Private Cloud with System Center 2012 and I ran into issues that I could not create a VMM connector to connect to my new VMM server from Service Manager. I got a very simple error message: “Cannot connect to VMM server <vmm server name>”
While I was troubleshooting it, I realised 2 other issues:
1. I couldn’t create a PowerShell remote session to my VMM server using PowerShell command like:
New-PSSession –ComputerName <VMM server FQDN> –Credential <Service Manager VMM Connector Account>
2. After I have created a connection to the new VMM server in the Orchestrator VMM integration pack, I modified an existing test runbook called “Get Cloud” to use the new connection, and it also failed.
All of these worked fine on the old VMM server, WinRM is configured exactly the same between old and new VMM severs because it configured in a domain GPO.
Both these errors were related to some kind of Kerberos authentication errors. (It was very late at night, I forgot to take screenshots while I was troubleshooting it).
After few hours troubleshooting, I have found the problems:
1. During VMM install, I installed SQL server on the VMM server. I somehow installed SSRS (SQL Server Reporting Services) as well although it’s not required for VMM. I had SSRS running on a service account and I have registered the SPN’s for SSRS (i.e. setspn.exe –A http/<VMM server Name> <Service Account>). I realised SSRS was installed by mistake, so I uninstalled it. As soon as I removed the SPN’s for the http service (for SSRS), PowerShell remote sessions started to work. – because PS Remoting also uses http.
2. I then tried to run the VMM PowerShell cmdlet Get-SCVMMServer in the remote shell against the VMM server and I got an error:
The type or name syntax of the registry key value IndigoTcpPort under Software\Microsoft\Microsoft System Center Virtual Machine Manager Administrator Console\Settings is incorrect.
Luckily I found this blog article. In my VMM server’s registry, the value IndigoTcpPort under HKLM\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Administrator Console\Settings is a REG_SZ (string) with no value configured. After I deleted it and recreated a REG_DWORD with value 1fa4 (which is hexadecimal for 8100), everything started working. The Get-SCVMMServer cmdlet worked fine. Service Manager VMM Connector was successfully created and Orchestrator runbooks were able to run.
So to summarise what I’ve done. I’ve reviewed and removed the SPN’s for http service which were originally created for SSRS, and corrected the “IndigoTcpPort” registry value.
It was not how I would like to spend my Saturday night, but I’m glad I’ve got it fixed so I can continue with my study.