Back to VMware

Over the past two months I have been working with Microsoft Virtual Server. When working on various numbers this year the licensing costs for VMware forced us to look at possible alternatives - one of these being Microsoft Virtual Server, with a possible move to Windows Virtualization when it is finally released.

Most of the experience with the Microsoft solution was very positive. Once the Virtual Additions were loaded in the Windows guest VMs I was able to get very good performance out of them. I ran a variety of servers on the host (which was running Windows Server 2003 R2 SP2 and the most recent Microsoft Virtual Server) and could run several VMs at once and still have good performance.

I was down to two final things to test when I finally ran into what we will consider a show stopper for any chance of us moving to the Microsoft solution. The first was to test a Linux guest VM and the second to test hot backups. Linux guests are now supported in the Microsoft solution and hot backups can supposedly be done through Microsoft Data Protection Manager using VSS. I never made it to testing hot backups, as the Linux experience proved to be the proverbial straw that broke the camel's back.

I installed a CentOS 4.x guest. The install went with no issues and it was easy to get setup. Wanting to use a SCSI disk I managed to get the Linux Virtual Additions to install. I wasn't very happy with this install though, mainly because they require you to use the --force switch to RPM to get the X portion of the extensions to install. In my opinion if you ever have to use the --force switch with RPM, something is wrong.

To get the SCSI portion to start at boot I needed to integrate it into initrd. This was not too bad and soon I had the vmadd-SCSI starting with no errors. Part of my testing was to test IO performance. I did this by running iostat in the background and using dd to create a 2GB file on the virtual disk. My first time doing this caused the guest to get numerous disk errors and render the Linux VM unusable. My other running Windows VMs did still run and respond to requests. However, when powering off the Linux VM it hung on shutdown. The only way I could find to power it off was to restart the host server. Luckily this was a test environment, but had this occurred in production it would have been unacceptable.

I tried more that afternooon to get the Linux VM to fail after I brought the host server backup. I could not get the Linux VM to fail though. When I came in the next morning though the Linux guest had died again with the same errors as seen the previous day. It had died at 8pm the previous night - the only thing going on at that time was the iostat command I left running refreshing every 2 seconds. Again, it required a reboot of the Virtual Server host to get the Linux guest to come back up.

Now, it is possible the actual issue within the guest could have been fixed. But, I had seen enough. There is apparently some condition that exists that can crash a guest VM bad enough that the whole server needs restarted. I cannot risk going into production when such conditions are found in testing.

I tested VMware ESX Server similarly when we considered it for production. I was never able to crash a guest VM to the point that the entire ESX server needed rebooted to resolve the issue. Powering off the guest always worked. Perhaps I just haven't run into the issue with ESX and there is some condition out there that will force me to reboot a production ESX box to fix one broken guest - but with the Microsoft solution I have actually confronted this issue in testing.

So we are back on track to continue our VMware ESX implementation. It feels good to move forward again. It also helps that VMware's new Foundation packs coming in December will really help smaller businesses like the one I work for afford the ESX products.

Busy News Days for Virtualization

It has been a busy couple of days in the virtualization realm of technology. Yesterday VMware had their IPO. Their stock did very well, increasing around 75%. Personally I expected this. They do make a very good product and are certainly the leaders in this arena for now.

Today Citrix announced their acquisition of XenSource for $500 million.

Meanwhile Microsoft continues to work on their virtualization products for the Enterprise with Virtual Server available now and Windows Server Virtualization coming 180 days after the release of Windows Server 2008.

Certainly very interesting times in the virtual world. It seems more and more companies are seeing the value in virtualization. Now it is just a matter of choosing which solution will work the best for you.

Microsoft's Virtualization Site Updated

As posted in a recent post in Matt McSpirit's blog the Microsoft Virtualization site has been updated. It looks like they are finally starting to bring all of the tools together under one section of the site to allow easy access to the tools, documents and resources related to Microsoft's Virtualization technology.

More SCVMM Fun

I have continued to do some more conversions with the tools in Beta 2 of the System Center Virtual Machine Manager. I did another V2V conversion of a Windows 2003 Server SP2 machine. That one went quite well -getting used to the Microsoft tools has certainly helped speed things up in getting a VMware VM converted to a Virtual Server VM.

I also did a Physical-to-Virtual (P2V) conversion today. The source server was a test server running Windows Server 2003 SP1. Unlike the V2V tools in the beta, the P2V tools can be driven through the GUI. After answering a few simple questions the conversion kicked off. I did not keep an exact eye on the time, but to convert the server running on an 18GB mirror it took around an hour for the conversion to finish I think. Add some more time to get the VM booted, new hardware detected and another reboot to get the Virtual Additions installed and it was a relatively quick process.

VMDK to VHD Conversion with Virtual Machine Manager Beta 2

I have been working with the conversion of VMware virtual machines to Microsoft Virtual Server machines. Microsoft has a public beta of System Center Virtual Machine Manager (SCVMM) (Beta 2) that includes Physical-to-Virtual (P2V) and Virtual-to-Virtual (V2V) capabilities. The conversion of physical machines is done through the GUI in SCVMM. Conversion of VMware's VMDK files is done through PowerShell commands.

I was most interested in playing with the V2V functionality of the tool to see how well it worked. I have several VMs in test and production running on various flavors of VMware's products. For my first attempt I worked with one of my smaller Windows 2000 Server with Service Pack 4 VMs.

I stopped the source virtual machine and copied its working directory to one of my Microsoft Virtual Server hosts. I am running SCVMM in a Virtual Server VM. The first step I did was to run the New-V2V command which has this description:

"Converts a virtual machine created in VMware(R) to a Virtual Server-based virtual machine managed by Virtual Machine Manager."

Working with a copy of the original VMware VM I followed the example from Get-Help New-V2V -detailed.

First I ran:

Get-VMMServer -Computername vmm-beta2.matrix.com

Per the notes in the example this retrieves the server object from the VMM database.

Then:

$VMHost = Get-VMHost -ComputerName "ghost.matrix.com"

This retrieves an object that represents the virtual machine host and allows me to reference that object as $VMHost.

and Finally:

New-V2V -VMXPath "\\ghost\c$\tmp\W2KServer_w-SP4_SPARKS\win2000serv.vmx" -VMHost $VMHost -Name "SPARKS01" -Path "e:\virt-servers\dump" -MemoryMB 256 -RunAsynchronously

This command points New-V2V to the vmx file for the VMware virtual machine and creates a VM on the Virtual Server VM Host in the Path listed. It also sets the amount of memory to 256MB. RunAsynchronously returns the shell prompt immediately while the job runs. You can watch the job in the admin console of SCVMM. There are also several other options detailed in the help file for New-V2V that you can pass regarding properties of the new Virtual Server VM.

This was a smaller VM that I was working with, only a 2.8GB VMDK file. The conversion went pretty quickly, but I was met with errors before completion.

"Error (3217)
Required files for the P2V conversion of \\ghost\c$\tmp\W2KServer_w-SP4_SPARKS\win2000Serv.vmx are missing from the P2V patch cache. The files are archived in DRIVER.CAB, which is located at %WINDIR%\Driver Cache\i386 on \\ghost\c$\tmp\W2KServer_w-SP4_SPARKS\win2000Serv.vmx or on the Microsoft Windows 2000 installation media.

Recommended Action
Copy DRIVER.CAB for the source operating system and paste the file to "C:\Program Files\Microsoft System Center Virtual Machine Manager 2007\P2V Patch Import" and then run Convert Physical Server wizard again."

This error also occurred for SP4.cab, the UPDATE ROLLUP 1 for SP4 and another update or two.

To resolve I copied the driver.cab and SP4.cab from other Windows 2000 Server SP4 machine to "C:\Program Files\Microsoft System Center Virtual Machine Manager 2007\P2V Patch Import" on the SCVMM host. I also downloaded the updates needed individually from the Microsoft site and copied them to the same location. Once those files are copied I ran the following command in the SCVMM PowerShell prompt to add them to the Patch Cache

PS C:\> $vmmserver = get-vmmserver -computername VM-BETA2
PS C:\> add-P2VPatch -vmmserver $vmmserver

Then, through the GUI I could click retry job to let the conversion process continue.

This portion of the conversion finished and I was able to start the new Virtual Server VM through the admin console for the SCVMM. The VM booted Windows 2000 with no issues. Upon logging in new hardware was detected and I was asked to reboot.

After the reboot I did lose my mouse and keyboard control for the VM. I could no longer enter CTRL-ALT-DEL to login. The keyboard did work to hit F8 to get to Safe Mode, but once in Safe Mode the keyboard and mouse did not work either.

At this point I decided to boot to the recovery console to see if I could find out what the issue was. First I noticed that VMware Tools was still trying to start. I disabled it from the Recovery Console by typing:

DISABLE vmwaretools

I rebooted after this and still had the issue. So back to the recover console I went. I noticed the i8042prt service was set to disabled. So I ran the following command to enable it:

ENABLE i8042prt SERVICE_BOOT_START

After a reboot the mouse and keyboard worked in the VM. I installed the Virtual Additions into the guest OS and all appears well with the newly converted VM.

COVUG Tech Days

I just got back from the Central Ohio VMware Users Group (COVUG) Tech Days at Kalahari Resort. It was a great time and very, very informative. The event was held up at the Kalahari Resort just a touch south of Sandusky, Ohio. The convention center and hotel were very nice - I believe the convention had approximate use of 10,000 sq. feet, so there was plenty of room for everybody.

The event had numerous vendors there with presentations and break out sessions held on both Thursday and Friday. VMware, HP, EMC, DataDomain, Vizioncore, esXpress, Equallogic, Dell, Intel, AMD, Acronis and others had booths and presentations.

I find these events great ways to get an overview of what is happening out there and act as a starting point for what I should be looking more into. Some of the things that caught my eye this time around...

I sat in on an Acronis presentation. It seems like a very nice product. The ability to restore on different hardware is very attractive and would be handy to have for any Virtual to Physical situations one may run into to make a vendor happy (i.e. to be on a physical box for support instead of virtual). I do plan on looking more into their product and see if it has a place in our organization.

I always try to sit in on Equallogic presentations and I managed to catch one this time as well. I still really like the hardware they put out. People are always saying good things about it and from the live demos I have seen it does seem to be a very impressive piece of equipment. I still lean towards getting one of these for Phase II of my virtualization project at work if the budget for next year will permit.

I managed to talk to David Siles, the CTO for Kane County who runs a lot of Equallogic storage with VMware. He is running to 5 9's availability and is running VMs, Exchange DBs and SQL DBs off his SATA based arrays. He had great things to say about the boxes and it was nice to hear from someone running some very serious workloads on just the SATA products.

I also sat in on an esXpress presentation. They do the backup software that does hot backups of VMs. I am using their software in production and am very fond of it. Their presentations are usually fun as the tech guys do them, not sales guys. They have a couple of cool things coming up soon - file level restores from within the VM and a feature to help allow automated replication to warm standby boxes.

I also attended several sessions done by VMware which were informative. Their NDA sessions are always a nice glimpse at what might be in store for us in the future.

All in all, a very successful trip. If you ever have the chance to attend one of the COVUG events I would encourage it. They are very informative and a good time.

Virtualization Hardware Purchase

After several months of planning and working out a plan for our company to make a move towards virtualization the PO's were signed and hardware is being ordered. The main server was ordered today along with our VMware licenses. A SAN is ready to be ordered - just have to cross paths with our local vendor to get those final details worked out.

We are moving ahead slowly and won't be taking full advantage of all that VMware has to offer. The primary goal is server consolidation at this point. The savings we get from knocking out a few of our underutlized servers should make management a little more open to further VMware investments with a Phase II hopefully coming next year. Phase II is where we build in the real redundancy and virtualize even more of our servers. Phase I just has us getting the low hanging fruit.

As for redundancy - we will only have one workhorse server running ESX. For Phase I though it will only run about 6 VMs or so. We will use a 3rd party backup product, probably esXpress, to hot backup the VMs to another storage location. In the event of a host failure one of the older boxes will be available to bring online and at least bring up some VMs to lessen the impact while the host server is repaired. While not the spiffy automatic roll over we could have had - it will allow fairly quick recovery by restoring those backups.

I am looking forward to the hardware getting there and getting some of this stuff setup! We've been using VMs in test for quite some time now. We have two VMs in production now, but the move to ESX Server will be the first major push for us. Should be fun!

VMware Server and SQL Server

I negelected to update this post regarding the issues we had been facing with Microsoft SQL 2000 running on VMware Server 1.0.1. Despite several attempts and different tactics tried I could not get the ERP upgrade the developer was working on to work without experiencing SQL errors.

I decided to try the eval copy of ESX Server. So I spent time shuffling off the VMs running on the current VMware Server (running CentOS 4.4 as the host) to my other one running on Windows. I then rebuilt the same hardware as an eval ESX 3.0.1 server. I moved the SQL VM back to the ESX server, made sure things were running good and turned it back over to the developer to test the ERP upgrade.

This time it worked flawlessly. The upgrade proceeded with no issues and completed. The developer has been happily working out the normal upgrade bugs with the ERP upgrade - which were not related the SQL issues at all, just broken customizations and such.

We are certainly glad it worked and I guess this just goes to reinforce that one should only run ESX Server in production.

VMware Server 1.0.2 Released

VMware released VMware Server 1.0.2 today, the release notes are here. I haven't had the chance to get it installed anywhere yet, but for those with more time it might be of interest.