Tag Archives: MimboloveHyper-V
Back in August / September last year, I spent sometime designed a set of Orchestrator runbooks to migrate Window Server 2008 R2 Hyper-V clusters to Windows Server 2012. I wasn’t going to blog this because it was designed to only cater for my company’s environment, not something that’s generic enough for everyone to use. I also wasn’t sure how well I can explain and document these runbooks in a blog post. Few of my colleagues and friends actually encouraged me to blog this one so I’ll give it a try (and not to disclose company related sensitive information).
My employer is one of the biggest two supermarkets and retail chains in Australia. Few years ago, there was a project that implemented a 2-node Windows Server 2008 R2 Hyper-V cluster in each of the 750 supermarkets that owned by my employer. More details about the implementation can be found from this Microsoft Case Study.
Early last year, we have started a project to upgrade the entire System Center suite to System Center 2012 (later on decided to go to R2). A part of this project is also to upgrade 2008 R2 Hyper-V clusters in supermarkets to Windows Server 2012 (and later on decided to go straight to Windows Server 2012 R2).
I have been in my current role for just over 2.5 years now, one thing that I learnt while managing a large retail environment is, that Automation is THE key to our success. No matter what solutions you are planning to rollout to the stores, if you can’t automate the rollout process, your plan is not going to fly. Therefore scripting is pretty much the essential requirement for everyone in my team. And in the office, you hear the phrase “cookie cutter” a lot. Automation standardise and simplifies implementation processes for new technologies. Our job is pretty much to design all kinds of cookie cutters and hand them over to other teams to cut cookies.
For the Hyper-V cluster upgrade, our end goal is that the implementers would enter a store number in the System Center Service Manager 2012 web portal and Service Manager would then kick off a series of Orchestrator runbooks to perform the upgrade. The upgrade would involve the following steps:
- Live migrates all VMs to cluster node 2.
- Evict Node 1 from the 2008 R2 Cluster and rebuild it to Windows Server 2012 R2 using ConfigMgr OSD.
- Migrate all cluster resources from the old cluster on Node 2 (Running Windows 2008 R2) to the new cluster on Node 1.
- Rebuild Node 2 to Windows Server 2012 R2
- Join Node 2 the the new cluster.
I have been asked to assist to design a prototype for step 3 using Orchestrator 2012.
Note: At the time designing this prototype, Windows Server 2012 R2 and System Center 2012 R2 was yet to be released and we had not made decisions to go straight to R2. Therefore my prototype was based on the scenario that we were going to migrate from Windows Server 2008 R2 to 2012 RTM. I had to make some modifications once we have decided to go to 2012 R2. I’ll explain it later in this article.
I firstly scripted the entire cluster migration process in PowerShell, using WinRM and Windows Server 2012 Server Migration Tool. Once the PowerShell script was working 100%, I then broke it up into many small segments and build the Orchestrator runbooks based on this script.
The purpose of this blog post to document my experience designing this prototype. I will provide the runbooks export at the end of this article.
In my prototype, there are 6 runbooks in total. The migration process would go from 01—>02—>03—>04—>05—>06:
01. Data Gathering
02. Prerequisites Checks
03. Export HyperV Config
04. Move Cluster Resources
05. Import HyperV Config
06. Add VMs to the Destination Cluster
These runbooks require the following prerequisites:
- The Orchestrator Runbook Service account needs to have local admin rights on both Hyper-V cluster nodes (this is because it’s only a prototype so I didn’t bother to setup proper service accounts for each activity).
- Hyper-V PowerShell module is installed on both old and new cluster nodes (Microsoft did not provide a Hyper-V PS module out of the box in Windows Server 2008 R2. So for 2008 R2, we used the one from CodePlex).
- WinRM is enabled on both cluster nodes.
01. Data Gathering
The purpose of the first runbook “Data Gathering” is to connect to both cluster nodes and gather few information for the subsequent runbooks. The only information that the operator needs to provide is the server names for both nodes.
It doesn’t matter which node is node 1 and which one is node 2, the runbooks will figure out the source and destination clusters by comparing the OS versions.
The very first step for runbook 01 is to check if the account it’s running under has local admin rights on both clusters. It will only continue of if it can connect to the admin$ share on both clusters.
This runbook will also get the FQDN, OS versions, WinRM ports from each cluster. With regards to WinRM port, because VMM 2008 and VMM 2012 use different WinRM ports when installing VMM agents on the Hyper-V hosts, so in order for other runbooks to connect to each Hyper-V cluster, we need to know which port to connect to (in 2008, port 80 is used and in 2012, it’s the default port of 5985).
02. Prerequisite Checks
This runbook performs the following checks:
01. OS Version Validation – Both cluster nodes cannot be on the same version, and the minimum OS major version must be 6.1 (Windows Server 2008 R2).
02. Cluster Validation – Make sure Windows Failover cluster role is installed on both nodes.
03. Identify Source & Destination Cluster – based on the Windows OS major version, the one with lower version is the source cluster, and the higher version one is the destination cluster.
04. HyperV Validation – Make sure Hyper-V role is installed on both cluster nodes.
05. Check Cluster Nodes Count – Make sure both source and destination clusters only have 1 node at this time.
06. Smig (Server Migration Tool) Feature Validation – Checks if Server Migration Tool is enabled on both clusters and if it is enabled, are both clusters running on the same version of Server Migration tool.
If step 6 (Smig Feature Validation) failed, Step 7 to Step 10 are performed to configure Smig on both clusters:
07. Create Smig Package on Destination Cluster – this step uses “smigdeploy.exe” on the destination cluster to create a smig package based on source cluster’s OS version.
08. Copy Smig Package To Source Cluster – This step copies the package on the destination cluster created in step 7 to the source cluster
09. Register Smig On Source Cluster – After step 8 has copied the package to the source cluster, this step registers the smig package on the source cluster.
10. Smig Feature Re-Validation – performs the same validation as step 6. by now, the server migration tool should be fully configured on both source and destination clusters.
11. Process Prerequisites Checks Result – consolidates results from each prerequisite checks. Continue to Runbook 03 if passed all prerequisite checks.
03. Export HyperV Config
This runbook exports the HyperV configurations on the source cluster using Server Migration tool and then copies the export to the source cluster. The following activities are performed in this runbook:
01. Shutdown VMs on source cluster – This activity shuts down all virtual machines that are currently running on the source Hyper-V cluster.
02. Remove Cluster Resources on Source cluster – because Server Migration tool does not support migrating clusters, so all virtual machine cluster resources needs to be removed from the source cluster.
03. Export HyperV Config From Source Cluster – now, none of the virtual machines are hosted on cluster, they are rather standalone VM’s (Not HA’d). we can now export Hyper-V configurations on the source cluster node using the Server Migration tool Powershell cmdlet Export-SmigServerSetting.
04. Delete Previous Export From Destination Cluster – if there is a copy of previously created Hyper-V smig export on the destination cluster, this step will delete it so we can then copy the most recent copy to destination cluster node next.
05. Copy Hyper-V Export to Destination Cluster – copy the export created in step 3 to the destination cluster node.
Note: for all file copy activities in my runbooks, I couldn’t run them within a Powershell remote session because I am not using CredSSP in my runbooks so I can’t connect to a remote UNC path within a PS remote session – because of the second-hop constraint. Therefore I’m simply running the robocopy command:
I remember I read somewhere that Orchestrator uses PsExec.exe under the hood when running a command against a remote computer. PSExec.exe uses RPC rather than WinRM.
06.Verify HyperV Export Copy on Destination Cluster – after the Hyper-V export has been copied to the destination cluster, this runbook verifies the export files by comparing the file count between the copies located on the source and destination locations. if this validation is passed, the next runbook is then triggered.
04. Move Cluster Resources
This runbook moves cluster resources (cluster shared volumes) to the destination cluster.
01. Delete VM’s on Source Cluster – This step deletes all virtual machines from the source node. at this point, the VMs are not hosted on the source cluster anymore, they are standalone non-HA’d VMs. they need to be deleted from Hyper-V now.
02. Move CSV’s – This steps moves all cluster shared volumes from the source cluster to the destination cluster. this is a tricky one. In order to keep track of the CSV names, I had to move only one CSV at a time. To move it, I firstly save the name of the CSV to a variable, then delete it from the source. after it’s deleted, I can then pick up this particular CSV as available storage in the new cluster. After I added the available storage in the new cluster, I rename the CSV to the name I captured from the source cluster. This is why we can only move one CSV at a time, if we move more than one, there is no way I can identify which one is which in the new cluster. For more information about this process, please refer to the PS script in this activity.
03. Delete CSV’s From Source Cluster – Now that the CSV’s are all moved to the destination, they can be deleted from the source cluster.
04. Delete Available Storage From Source Cluster – After the CSVs are deleted from the source, they are still showing as available storage in the source cluster. This is VERY IMPORTANT, these available storage MUST BE deleted. Without this step, Server Migration Tool will not work 100%. I got stuck here for few days when I was designing the runbooks. Server migration tool does not like clusters. without this step, I was keep getting NTFS related errors when importing the Hyper-V export to the destination cluster using Server migration tool. but if I ran the import again after the first attempt failed, it would work second time. After I added this step in the runbook, the problem went away. so in another word, everything has to be completely removed from the source cluster before we can import the Hyper-V configuration to the destination cluster.
05. Import HyperV Config
now that we have all the darts lined up, it’s time to import the Hyper-V configuration to the destination cluster. This runbook only contains one activity:
01. Import HyperV Config on Destination Cluster – the script in this step firstly make sure all CSV’s are online, then perform the import using Import-SmigServerSetting cmdlet from Server Migration Tool PS snap-in. The HyperV export created from the source cluster also contains information such as Hyper-V virtual switches. after I imported the config to the destination cluster, I do not have make any additional configuration changes as Server Migration tool took care of everything.
06. Add VMs to the Destination Cluster
Now that all VMs are migrated to the destination cluster node. this runbook add them to the destination cluster and starts them up. This runbook contains 2 steps:
01. Add VMs to Cluster – This step adds all VMs to the cluster.
02. Start VMs – This steps powers on all VMs.
Test Drive Results
when we tested this prototype in our lab, with 3-4 VMs hosted on the cluster, These runbooks took around 10-15 minutes to run. – And all the wait I hardcoded in my scripts contributed to this time frame as well. It is completely automated, all we did was to enter the NetBIOS names of the 2 cluster nodes to kick off the first runbook, and sat back, watching Orchestrator to progress with each activity.
Story about Windows 2012 R2
So I finished the prototype by the end of September last year and went on holidays for 4 weeks. Windows 2012 R2 was released while I was on holidays. After I came back, my colleague told me these runbooks didn’t work when migrating from Windows Server 2008 R2 to 2012 R2. because Server Migration Tool only supports migrating from n-1 version when come to Hyper-V. When creating the 2012 R2 version of the smig package for 2008 R2, Hyper-V is not available. However, migration from Windows Server 2012 to 2012 R2 still works using this runbook prototype (because it’s only 1 version behind).
Microsoft advised there is an unsupported undocumented way to make Hyper-V available in the package but we have decided not to use this method because it’s not supported.
After logged a support call with Microsoft and they have advised to just import the Hyper-V VM’s (the xml) to the new cluster. So I spent another couple of hours to modify the existing runbooks to ditch Server Migration Tool and directly import the VM’s XML into Hyper-V. This has made the runbooks simpler than before, but the drawback is that other Hyper-V settings such as virtual switches are not migrated across. Luckily since we have cookie cutters for everything, it’s a no brainer for us to make sure the Hyper-V host servers base build includes the virtual switches creation.
What happened next?
I handed the prototype to some other people and they have then spent few months enhancing it and built the rest of the store upgrade / conversion process based on this prototype. I can’t really disclose too many details about this (at least not at this stage). It has now grown to a monster with many runbooks and integrated with few other System Center 2012 products.
The purpose of this post is to share my experience of designing this specific “cookie cutter” prototype. I hope this would help someone when performing similar kind of tasks.
Both versions of the runbooks can be downloaded from the links below:
Note: I have removed all the email activities from the runbooks exports from the links above.
Just another quick note about the runbooks – When I created these runbooks, I created a folder called “HYPER-WEE”, as a joke. After all the runbooks were finalised, I realised it’s too hard to rename this folder because it’s hardcoded in each Invoke Runbook activity:
I didn’t bother to rename the folder and update all the runbooks. So if you have downloaded and imported the runbook, you’ll see it:
Lastly, as always, please feel free to contact me for any questions or issues.
I’m not having any luck with computers lately. This is why all my recent posts are related to my troubleshooting experience.
Yesterday morning, there were some tradies working in my house fixing a leakage in my shower. I was just about to get in the car leaving home for the MS 70-246 exam, they connected some tools to a powerpoint and popped the safety switch. Therefore all my computers got shutdown unexpectly. I quickly switch the safety switch back on and left home.
After I came home few hours later, I turned all the computers back on and made sure all VM’s were up and running.
Today is a public holiday in Australia and New Zealand, and since I’ve just passed a exam yesterday after 2 months of study, I was going to spend some time with my daughter and trying to stay away from doing more work in the lab.
Unfortunately, my day didn’t turnout as what I’ve planned. This morning, I noticed there were around 15 VMs missing from one of my Windows Server 2012 Hyper-V hosts. I had 21 VMs running on that box and I could only see 6. all the rest were gone from the Hyper-V console.
On this Hyper-V server (named HYPERV01), there are 3 SATA disks and 2 SSDs hosting VMs. I found out all VMs from 2 out of 3 SATA drives went missing.
I checked C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines and all the symbolic links for VM’s are still there. Below errors were logged in the event log:
According to http://support.microsoft.com/kb/2249906, It could be caused by insufficient NTFS permissions. I double, triple checked NTFS permissions, and used icacls command to assign virtual machine SID permission to VHDs and XMLs (as suggested in the KB), it didn’t help.
Another KB, http://support.microsoft.com/kb/969556 suggests it’s caused by Intel IPMI driver. Although I’m running Windows Server 2012, not 2008 R2, I still downloaded Intel’s ResetAccess utility as suggested in the KB but it couldn’t run on Windows Server 2012.
KB961804(http://support.microsoft.com/kb/961804) reckons it’s caused by antivirus on access scan and exclusion list. I’m using SCEP (System Center Endpoint Protection) which comes with SCCM 2012. I previously configured exclusion using the Hyper-V antimalware policy template that comes with SCCM, and on top of what’s in the template, I’ve also added few file extensions to the policy (.VHD; .VHDX; .AVHD). However, I didn’t add .XML file type and the path to where VM’s are stored. I couldn’t go to SCCM and fix up the policy – because the Central and the primary site server where my Hyper-V is reporting to are among those 15 missing VM’s.
I didn’t configure to allow users to override SCEP on-access scan settings and exclusion list. So there is no way I could configure SCEP. I’ve then uninstalled SCCM client using “ccmsetup /uninstall” and and uninstalled SCEP agent from Programs and Features in Control panel. I rebooted HYPERV01 after the uninstallation.
After reboot, nothing’s changed. Still not fixed. I then spent next 5-6 hours tried many things including copying all VM’s out of a problematic drive and the reformatted the drive…
I also found in VMM console, the 2 problematic disks (D:\ and E:\) did not show up in the Hyper-V server properties:
Under the Storage tab, it showed these 2 drives have 0 GB available but in fact, both of them only have around 200 GB data on the 1 TB drives.
I tried to re-import the missing VM’s back, but I got this error:
Finally, at 11:00pm, I managed to fix the issue. I uninstalled “Windows Firewall Configuration Provider” from Programs and Features
According to some Google search results, it is a part of SCEP client. After uninstallation and a reboot, all my VM’s appeared in the Hyper-V console:
For those VMs that were copied back to the formatted drive, I had to configure NTFS permission as per KB2249906 before they could be started (because during the copying processes, the VM SID has lost access). (Tip: use /T switch in icacls command to apply the permission to all files and folders below).
Now I’ll have to put SCCM client back on and re-configure Hyper-V antimalware policy. I’ll leave it to tomorrow…
Anyways, this is how I spent my ANZAC day holiday. Maybe it’s time to get few UPS’s for the lab…
Part 1 can be found here.
The content of this article is purely based on my personal experience and opinions. I have absolutely no intentions to criticise vyatta. To be honest, I still think it’s a great product, however, it just does not suit my needs in my lab environment.
Stefan Stranger has written a great article on Vyatta Virtual Router on Hyper-V back in 2008. The version Stefan used in his article was 4.0 and when I am writing this article, the latest version is 6.4. It can be downloaded here: http://www.vyatta.org/downloads
Vyatta is extremely light-weight. In my previous environment, I only needed to assign 256MB of RAM to each Vyatta instance. It is also very easy to setup. for me to configure an instance from scratch, would take me no more than 10 minutes.
Below is a list the setup commands I had to run to configure a Vyatta from scratch (based on my previous lab configuration):
set system host-name vyatta
set interfaces ethernet eth0 address 192.168.1.254/24
set interfaces ethernet eth1 address 192.168.2.254/24
set interfaces ethernet eth2 address 192.168.3.254/24
set interfaces ethernet eth3 address 192.168.6.254/24
set service ssh
set service telnet
set system name-server 192.168.2.10
set system name-server 192.168.4.10
set system gateway-address 192.168.1.1
set protocols static route 0.0.0.0/0 next-hop 192.168.1.1
set service snmp community public
set system login user vyatta authentication plaintext-password password1234
So why am I moving away from Vyatta? the short answer is: Vyatta does not officially support Hyper-V:
Read between the lines, Hyper-V is not supported.
What does it mean in my lab Hyper-V 2008 R2 hosts in the past (Based on version 6.1 that I have implemented)?
- After I reboot the Vyatta VM, all configurations were lost. It needed to be reconfigured again from scratch (that’s why I became so good at its commands). To work around this issue, I created a VM snapshot after it’s fully configured and I had to revert it back to the snapshot after every reboot.
- It does not support Hyper-V Synthetic NICs. This means I’m stuck with legacy NICs for Vyatta. Legacy NICs means 100mb/s instead of 10GB/s and I can only assign maximum 4 NICs to the Vyatta instance. This is why I’ve only got 4 virtual switches configured for each Hyper-V host in my lab.
- Vyatta is a cut down version of Linux, I could not install Linux Integration Service for Hyper-V. Otherwise, I might have already fixed the above mentioned 2 issues.
- Besides myself, two of my colleagues also tried to “Install” Vyatta 6.4 on the VHD (as oppose to run the Live CD mode). All 3 of us had the same issue: it runs OK, but the network latency caused by Vyatta is unacceptable. The ping latency from subnet A to subnet B in the same Hyper-V host gets to 2-4 seconds (2000-4000ms).
My colleague Matt McGowan spent sometime over a weekend and tried all versions of Vyatta after version 6 on a Hyper-V 2012 server, according to him, none of them could even boot up without disconnect all the legacy NICs first. This has become the last straw for me to give up on Vyatta. I had to find a better solution for my Hyper-V environment.
At that time, I was seriously thinking about buying a layer 3 managed switch (HP 1910-16G), which costs $400 AUD in my local computer shop. In the end, I’m glad I didn’t. After spoken to my good friend Zheng Han who is a RHCE and VCP, he advised me to take a look at CentOS.
So long story short, I’m hopeless when comes to Linux / Unix. I haven’t really done much with it ever since I graduated from uni. After a week or so playing with the latest version (CentOS 6.3) and learning how to use “vi”, with Zheng’s help, I got it working.
You’ve probably already seen the network diagram for my lab from Part 1. Here’s a logical diagram for the 3 Hyper-V hosts and the CentOS router in each host:
The Visio diagram of the above view can be downloaded here.
Now, I’ll use HyperVRT01 (the router on HyperV01) as an example and go through the steps of setting it up.
The following software is required:
- CentOS 6.3 (CentOS-6.3-x86_64-bin-DVD1.iso) –can be downloaded from a mirror site near you.
- Linux Integration Service For Hyper-V 3.4 (LinuxICv34.iso) – download from Microsoft.
Other Network information:
I have 2 domain controllers in my lab domain:
DC01 is also configured as a DHCP server serving multiple scopes in my lab. You’ll see the IP address of these 2 machines a lot in below steps.
*Note: if you are going to use CentOS and you are like me, a Linux noob, before you start, please make sure you get familiar with “vi” editor because it is heavily used.
Now, let’s start…
1. Create a new virtual machine in Hyper-V with the following settings:
- CPU: 1 virtual processor
- Memory: 512MB static
- Hard disk: 10GB (after install, I checked and only used 3GB)
- Assign 4 network adapters (note, DO NOT use legacy network adapters):
- Assign the 4 network adapters to virtual switches IN ORDER:
- #1: 192.168.1.0
- #2: 192.168.7.0
- #3: 192.168.8.0
- #4: 192.168.9.0
2. Mount the CentOS-6.3-x86_64-bin-DVD1.iso to the VM
3. Power on the VM to start installing CentOS
- Choose “Install system with basic video driver”
*Note: if you choose the first option to use the GUI based install wizard, you’ll need to assign minimum 1GB of memory to the VM. GUI based install won’t run on 512MB RAM.
- the rest of the install process is pretty much fool proof. I won’t waste my time going through the entire CentOS 6 install here.
Assume now CentOS is installed.
4. Install Linux Integration Service for Hyper-V:
- Mount LinuxICv34.ISO to the guest OS (HyperVRT01)
- Use the following command to install:
|mount /dev/cdrom /media
5. Disable Firewall
|service iptables stop
chkconfig iptables off
6. Configure DNS
|echo “nameserver 192.168.4.10” >/etc/resolv.conf
echo “nameserver 192.168.2.10” >>/etc/resolv.conf
7. Network settings
vi /etc/sysconfig/network then insert
8. Set IP Address:
- Start up all NICs:
|ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
ifconfig eth3 up
vi /etc/sysconfig/network-scripts/ifcfg-eth0 then insert
*Hint: Configure the first NIC eth0, then you can use Putty to connect via SSH. once in Putty, you can copy & paste commands.
vi /etc/sysconfig/network-scripts/ifcfg-eth1 then insert
vi /etc/sysconfig/network-scripts/ifcfg-eth2 then insert
vi /etc/sysconfig/network-scripts/ifcfg-eth3 then insert
- Restart network service
|service network restart|
Make sure all NICs start up OK:
9. OS Update
10. Enable IP forwarding (Routing)
- Check if routing is enabled:
0 = disabled
1 = enabled
- To enable routing:
vi /etc/sysctl.conf Then Edit:
|net.ipv4.ip_forward = 1|
11. Configure Route:
vi /etc/sysconfig/network-scripts/route-eth0 and insert:
|192.168.2.0/24 via 192.168.1.254 dev eth0
192.168.3.0/24 via 192.168.1.254 dev eth0
192.168.4.0/24 via 192.168.1.253 dev eth0
192.168.5.0/24 via 192.168.1.253 dev eth0
192.168.6.0/24 via 192.168.1.254 dev eth0
*Note: above list represent all subnets in the other 2 Hyper-V servers in my lab.
12. Restart Network Service again
|service network restart|
13. Configure DHCP Relay
- Install DHCP
|yum install dhcp|
- Configure DHCP Relay service (dhcrelay)
vi /etc/sysconfig/dhcrelay and Modify:
|INTERFACES=”eth0 eth1 eth2 eth3″
- Start DHCP Relay service (dhcrelay)
|chkconfig dhcrelay on
service dhcrelay start
14. Configure SNMP (Optional)
- Install SNMP:
|yum install net-snmp-utils
yum install net-snmp
- Backup SNMP Config File
mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.org
- Create new config file :
vi /etc/snmp/snmpd.conf and insert:
syslocation “TYANG.ORG Head Office”
- Start SNMP service
|service snmpd start
chkconfig snmpd on
15. Install Webmin (Optional)
- Install the GPG key
|rpm –import http://www.webmin.com/jcameron-key.asc|
- Add webmin repository
vi /etc/yum.repos.d/webmin.repo and add:
name=Webmin Distribution Neutral
- update the repos:
- Install webmin
|yum install webmin|
- access webmin page:
This is it. the router is now setup!
After setup, check system status via webmin:
As you can see, after everything is configured, it only uses 136MB of memory and 3GB of disk space.
Routing and Gateways:
Additional, I made sure a windows client OS could obtain an IP address from the DHCP server (which is located on another Hyper-V host). and I was able to ping / trace route other VMs in other Hyper-V servers (I have already demonstrated it in part 1).
This concludes this 2-part series. Before I go, I have to reiterate, there is nothing wrong with the Vyatta product, it’s just it does not integrate with Hyper-V too well. Unlike Vyatta, CentOS 6 is a fully supported guest OS in Hyper-V (With Linux Integration Service for Hyper-V 3.4) and CentOS 5 and 6 are also supported in SCOM 2012 SP1 beta! Having said that, Vyatta was running perfectly fine in my VMware workstation previously. If Vyatta adds support to Hyper-V in the future, I would definitely consider it again.
Lastly, please feel free to get in touch with me if you believe there are anything inaccurate in this series or you need more information in regards to the CentOS router setup.
I started a lab environment at home around 2 years ago and I’ve been keep investing in this lab environment over the time. I often get asked how was my lab setup. And most of the time when I tried to answer the question and explain how it was setup, I often forget some components. I have been wanting to properly document it for a long time but never got around to it.
Until 2 weeks ago, I had 2 machines with 24GB RAM each running bare-metal Hyper-V 2008 R2 and another desktop with 8GB RAM running Windows 7 with VMware workstation on top which hosts couple of VMs. The capacity of the lab was running very very low ever since System Center 2012 was released earlier this year as I had to run both 2007 and 2012 environments at same time. I had to down size my 2007 environments and decommission some machines such as my Exchange 2010 box to free up some capacity.
2 weeks ago, I bought another machine with 64GB of RAM and started rebuilding the entire lab with Windows Server 2012 and Windows 8. I though it’s a good time to document and blog how the lab is setup so I’ve got my lab configuration documented and I can simply point people to this post when I get asked next time.
This topic is probably going to be too big for one post so I’m going to split it into a 2-part series.
I’m going to concentrate on the physical / hardware part of the lab in part 1.
Ever since my lab was initially setup, I’ve been using vyatta virtual router appliance on the Hyper-V 2008 R2 and VMware workstation machines. During the rebuild process over the last 2 weeks, I have spent a lot of time working / discussing the routing solutions for the lab with colleagues and friends and finally implemented another what I believe a better routing solution using CentOS 6.3. In part 2, I’ll discuss my previous experience with vyatta and go through detailed steps how I implemented CentOS as virtual routers.
- In this 2-part series, I’m going to touch base on Hyper-V, networking, Linux and computer hardware, which none of these that I consider myself an expert of. There might be some parts of the article that you consider to be inaccurate or there are better ways of doing it. I’m always open for suggestions and constructive feedback. At the end of the day, all of this is about sharing my personal experience with the community and documenting what I have done so far for myself.
OK. let’s start.
First of all, every piece of hardware in my lab is considered consumer grade product. which means all machines are built using PC components and there are no server computers, no UPS, no rakcs, no layer 2 / layer 3 switches, no fibre channel cards or switches, no shared storage (either iSCSI or fibre), no tape libraries…. – Funny enough, I know someone got them all at home! I’ve always been very conscious of what to put in the lab. I had an old HP DL380 server which I ended up gave away because I can’t stand the noise it generates in my study!
Secondly, All the Microsoft software I use are licensed through my TechNet subscription. For around $400 AUD a year, I get to use pretty much everything I need from Microsoft (almost everything, Visual Studio and SQL Developer edition is not covered in TechNet subscription). I know that in other countries, TechNet subscriptions are significantly cheaper than Australia. i.e. the price in USA is almost half of what we pay here! It’s good investment. not to mention you get your money back during tax return every year anyway.
Above is roughly how everything hangs together at home. The original Visio diagram can be downloaded here if you can’t see the details in this small picture.
- OS: Windows Server 2012 Data Center With Hyper-V Role
- Motherboard: Intel Desktop Board DX79SR
- CPU: Intel Core i7 3820
- Memory: 64GB (8x8GB DDR3)
- Hard Drive: 3x1TB SATA and 1x120GB SSD
- NIC: 2 Onboard Intel GB NIC and 1 Intel PCI GB NIC
- OS: Windows Server 2012 Data Center With Hyper-V Role
- Motherboard: ASUS P6X58D-E
- CPU: Intel Core i7 950
- Memory: 24GB (6x4GB DDR3)
- Hard Drive: 3x500GB SATA and 1x120GB SSD
- NIC: 1 Onboard Marvell GB NIC and 2 Intel PCI GB NIC
- OS: Windows 8 Enterprise With Hyper-V Role
- Motherboard: ASUS P6X58D-E
- CPU: Intel Core i7 950
- Memory: 24GB (6x4GB DDR3)
- Hard Drive: Too many to count. 16TB in total
- NIC: 1 Onboard Marvell GB NIC and 1 Intel PCI GB NIC
- OS: Windows Server 2012 Enterprise
- Motherboard: ASUS P5Q Deluxe
- CPU: Intel Core2 Quad Q9400
- Memory: 8GB (4x2GB DDR2)
- Hard Drive: 1x500GB SATA
- NIC: 2 Onboard Marvell GB NIC
ADSL Modem & Router with WiFi AP and 4 port switch
Home & Lab Network 8-Port GB Switches
- 2 x NetGear GS608
Lab Network 5-Port GB Switch
- 1 x NetGear GS605
Lab Wireless Access Point
So above list is pretty much what’s been used in the lab. I’ve also got 2 Windows laptop and a Macbook Pro that I can connect to the lab environment either by physically connected to the 5-port switch or via the lab WiFi. the media centre PC sitting in the rumpus room is also member of my lab AD domain and is being managed by my SCCM and SCOM infrastructure in the lab.
Now that since I’ve mentioned my media centre PC is being managed by System Center, I’d like to remind you that if you want to do the same at home and use SCCM to patch your media centre machine, please keep in mind DO NOT set the SCCM maintenance window for this machine to be every Friday or Saturday night! I was once watching a movie with my wife on a Saturday night and a message box popped up saying the machine is going to be patched and I couldn’t cancel it because it has passed the deadline of the update deployment! In the end, we had to wait for it to complete… It made me look stupid in front of my wife…
Anyways, now let me explain the setup in details.
Motherboard & Memory
The original 2 Hyper-V hosts were running on the ASUS P6X58D-E motherboard with 24GB (6x4GB) of RAM. I bought these 2 machines back in 2010 and 2011. Back then, I couldn’t find any PC motherboard that has more than 6 memory slots and the biggest PC memory stick I could get in the computer shops were 4GB sticks. Therefore the 2 PCs with this ASUS motherboard have 24GB each. In the ASUS website, it stated the maximum memory this board supports is 24GB. which is based on 6 slots x 4GB. I have actually tested 8GB sticks on this board and it worked just fine. so when I’m running out of capacity again in the future, I can upgrade the RAM on these board to 48GB (6x8GB).
I have to say, I’m really impressed with the new motherboard that I’ve just got: Intel DX79SR. it has 8 memory slots, 4xUSB 3 ports on the back panel and 1 for the front. 2xIntel onboard GB NICs, 4x6Gb/s SATA connectors, 4x3Gb/s SATA connectors. It even comes with a WiFi/bluethooth module which you can optionally attach to the case (connected to a USB 2 connector on the motherboard). It has pretty much everything I need for the Hyper-V box. There’s no need to connect the WiFi and bluetooth module to the Hyper-V server. Since it’s connecting to the motherboard using a USB port, I’ve connected this to my desktop machine:
I’ve got 3 traditional SATA drives on each Hyper-V host to spread the Disk I/O for virtual machines. I’ve also added a 120GB SSD for each host to host VHDs used for SQL databases. for example, when I created a VM to run SQL for SCOM, I’ve created 2 separate VHDs on the SSD drive to host the SQL data and log files.
None of these motherboards have onboard video. I just bought whatever the cheapest NVidia card I could find for each machine. At the end of the day, I RDP to these machines 99% of the time and I don’t play with RemoteFX so there’s no need to utilise the GPU on servers.
In HyperV01, I’ve configured 4 virtual switches. one virtual switch (192.168.1.0) is connected to the physical network using a teamed connection (Now Windows Server 2012 supports NIC teaming natively). The rest are internal switches:
In Hyperv02, it’s very similar to Hyperv01. However I have 2 external connections. 192.168.1.0 is my physical network shared with the rest of home network. 192.168.6.0 network is connected to a physical NIC which is connected to the 5-Port Lab GB switch. I’ve configured it this way so I can physically connect my laptops to the lab using this connection and play with PXE boot and OSD in SCCM . I’ve also connected a WiFi access point to 5-port switch so I can connect the laptop wirelessly.
*Note: At the time of writing this article, Hyperv02 is still running bare metal Hyper-V 2008 R2. thus above screenshot looks a little bit different.
Another reason why I added the 5-port switch and WiFi access point to the lab environment is because the issue with name resolution.
My ADSL router is also configured as a DHCP server serving other devices at home (i.e. tablets, phones, Xbox, Wii and laptops when they are not connected to the lab). The DNS servers configured in the ADSL router DHCP scope are pointed to my ISP’s DNS servers. I cannot set conditional forwarding on my ADSL router to forward DNS requests for my lab machines to the domain controllers in my lab. I don’t really want to use my lab DCs for name resolution for all other devices at home because it means I’ll have to make sure these 2 DCs I have in the lab have to be always on. Therefore, to be able to login to the lab domain, the DNS settings on the machine has to be statically configured (which I’ve done for my media centre PC), or the IP configuration has to come from the DHCP server in my lab (which is one of my DCs). Therefore, I’ve added the 5-port switch and WiFi access point in the lab environment.
In my study PC, I’ve only configured 2 internal networks:
So all up, I’ve got 9 subnets: 192.168.1.0 – 192.168.9.0
Now, here is where the virtual routers come in. I need to be able to route the network traffic across each virtual network inside virtual hosts. I don’t have lay 3 switches or routers at home to allow me to do so.
In each Hyper-V host, I’ve also configured a virtual router device (I used to use vyatta but now I’ve switched to CentOS. This is going to be covered in part 2). Take Hyperv01 for example, there’s a virtual machine called HyperVRT01, which is the virtual router for this host (running CentOS):
For each virtual switch on HyperV01, I’ve configured a Network Adapter for the router HyperVRT01:
Since I have 3 virtual hosts in total, I now have 3 CentOS instances running, each of them has been configured one connection to 192.168.1.0 (which is my physical network). each of these 3 connections has been given an IP address:
HyperVRT01 (hosted by HyperV01): 192.168.1.252
HyperVRT02 (hosted by HyperV02): 192.168.1.254
StudyRT01 (hosted by Study): 192.168.1.253
these 3 IP addresses becomes the ultimate gateway for all virtual networks inside the Hyper-V host to the outside world.
I then configure the static routes within CentOS (again, to be explained in Part 2).
Additionally, I’ve configured static routes in my ADSL router:
This allows the rest of the home network to communicate to the lab (via IP address).
*Note: The network speed on the 4-Port switch on the ADSL router is only 100MB. However, since I’ve configured the static routes in each CentOS instance, traffic between each Hyper-V host does not go over the ADSL router (192.168.1.1). i.e. if I trace route from a VM from HyperV01 to a VM in Study:
As shown above, the traffic went straight from one CentOS router to the other CentOS router.
I often need to access my lab when I’m not at home (i.e. in the office). This is pretty easy to configure. I firstly register a dynamic DNS name from a provider supported by my router (i.e. dyndns). I then configure port mapping on the router. i.e. external TCP port 3390 points back to port 3389 of a VM in the lab (3389 = RDP port). or external TCPport 8888 to port 22 of a CentOS instance (22 = SSH port). When I’m in the office, I use my laptop with a wireless boardband device to connect back to home lab environment.
If you are working in IT infrastructure space, you are probably that’s it’s common for organisations to keep track of their IP address allocations. Since day one, I’ve been using an Excel spreadsheet to track how are IP addresses used in my lab. In the spreadsheet, I’ve create a sheet (tab) for each subnet. it’s been working great for me. If you have a complex environment and you don’t have a way to track your IP addresses, I’d suggest you to do the same!
OK, this is pretty much all I have for part 1 of the series. In part 2, I’ll go through the background and experience on why I went away from vyatta and steps I setup each CentOS instances as virtual routers. Stay tuned!
05/10/2012: Continue to Part 2!
I’m currently running Hyper-V R2 on a machine with 24GB of memory at home. It hosts most of my test machines such as SCCM, SCOM, Exchange, etc. System Center Virtual Machine Manager 2008 R2 (VMM) is installed on a separate box to manage this Hyper-V Host.
Few days ago the Hyper-V machine was powered off unexpectly and when it powered back online, in VMM, the status of 2 virtual machines showed as “Missing”.
I checked the location where virtual machines are stored and without doubt, the vhd, xml and other files for each virtual machine are still there.
I Checked the Event Log and found a lot of Event ID 16310 events in Hyper-V-VMMS logs:
The virtual machine ID in above event belongs to my SCOM server, one of the “Missing” virtual machines.
I googled it and some people somewhere had similar issues and it was because the VM configuration XML file was missing the closing tag “</Configuration>” or there were few additional lines after “</Configuration>”.
In my case, the XML file looks perfectly fine, BUT there is an additional space(“ “) after </Configuration>:
After removing the additional space and restarted Hyper-V Virtual Machine Management service (net stop vmms && net start vmms), the status of my SCOM server is “Stopped” and I was able to successfully start it!