Upgrading Hosts

VMWare has made it very easy to upgrade hosts to ESX 4. I made use of two remote upgrade options in my migration:

vCenter Update Manager

A centralized update management system which can handle anything from security updates to full ESX upgrades

The Host Update Utility

A standalone application built into the new vSphere client

vCenter Update Manager

For the local host machines, I made use of vCenter’s update manager. This is an add-on for vCenter and it requires its own database, but I highly recommend using it. Its end functionality is similar to any built-in software update mechanism, but its flexibility is what makes it really impressive. For information on installing, please see the administration guide.

Update Manager works on the idea of a “baseline”: a set of minimum requirements that a target needs to meet to be considered compliant. Baselines can be applied at any level in the vCenter hierarchy. To handle the upgrade of our host machines, I created a baseline for ESX hosts specifying that they should be at level 4.0. I then applied this baseline at the datacenter level in both sites. In some cases it may be more apt to apply it at the cluster level, but we have no necessity for legacy hosts in our case.

Once the baseline was applied, I selected the host I wished to update and clicked the update manager tab. I could see, as expected, that it was not compliant with the upgrade baseline applied to it. In a DRS cluster, one should be able to click the remediate button and let vCenter do all of the work to migrate VMs off and put the host in maintenance mode prior to the upgrade. This requires that you keep a pretty clean environment, however.

In our case, we had some hosts that were unable to vMotion either due to snapshots or mapped drives, so I had to take care of things manually. As soon as the host was in maintenance mode, however, choosing remediate rebooted the host and performed the upgrade all on its own within about 20 minutes or so. The progress bar was fairly accurate at most points of the upgrade and once the host was back up, I could see that it was compliant with the baseline.

The Host Update Utility

One major concern in my case was a couple of hosts sitting on the other side of a slow (3mbps) WAN link. I’m sure you’ll agree that most IT tasks are at their best when they go unnoticed. Well, taking up a huge chunk of a site’s bandwidth while I transferred an ISO multiple times would certainly not win me any gold stars.

VMWare’s Update Manger is capable of staging updates on the target for later install, but this only works on hosts that are at 4.0 or above. It would also require moving the ISO over the WAN multiple times, because the updates are staged to the host rather than another vCenter or Update Manager server. This made the Host Update Utility a much more interesting option.

Fortunately, there’s already data replication going on between the main site and the remote one for DR purposes. I was able to mount a snapshot of one of our DR volumes and copy the ISO onto a file server in that site. Launching the host update utility for the first time, I received a prompt to download patches from VMWare. The servers then needed to be added by hostname from the file menu. Once they’re added, the host update utility will do a quick scan to find out what version they’re at. If it’s in an upgradable state, the “Upgrade Host” button will initiate the upgrade.

Overall the procedure from this point on is very similar to the update manager service. The one notable exception is that there is a lot less feedback regarding the status of the host. There were a few times where I was wondering if the hosts were stalled, but eventually the progress bar jumped forward and I could see that things were still working. For this reason, I definitely recommend using the update manager service where possible.

Upgrading VMs

To take full advantage of the new functionality in vSphere 4, the VMs, themselves, must be upgraded. ESX 3.x operated on virtual hardware version 4. vSphere uses version 7, which has a number of notable improvements in storage, networking, and hotswap capability. It’s important to note that there are a few disappointments with the new networking and storage modules, but overall it’s a large step forward. There is a useful article at boche.net that talks about memory and CPU hotplug.

Upgrading the VMs is a two step process requiring at least 2 reboots (my process includes 3). First, the VMWare tools installation must be upgraded in the guest operating system. This may be done either from the vSphere client or from within the guest OS by opening the VMWare tools console. In my case, I chose to upgrade from within the guest so I was keeping an eye on every step of the process. This may not be viable for anyone with a large number of VMs to update. Upgrading the VMWare tools will require a reboot.

Secondly, the virtual hardware needs to be upgraded. The virtual machine needs to be powered down for this step. It’s a simple matter of right-clicking the virtual machine and selecting “Upgrade virtual hardware”. A message will pop up notifying you that this is a one-way process and once it’s finished, your upgraded VM will not work on older versions of ESX. Keep this in mind if you have any legacy hosts staying in your environment.

The next time the guest OS boots, it will detect the new hardware and configure it. For Windows VMs, this will require another reboot. I haven’t, yet, tested the process on a Linux VM. Once the guest OS is finished rebooting, the upgrade process is complete.

Environment

Our environment is fairly small in terms of number of hosts, but we have most of the components in place, which makes for a very nice upgrade experience.

• A VirtualCenter Server v2.5 (primary data center)
• Two host machines v3.5 (primary data center)
• Two host machines v3.0 (disaster recovery data center)

Procedure

The first thing that needs to happen during the migration is upgrading the VirtualCenter server. vCenter 4 is very capable of managing legacy products, but it’s a good idea to take a look at the compatibility matrices just to be safe.

Migration Strategy

The hardware that our current VirtualCenter Server was running on was less than adequate for running vCenter 4. So, I went ahead and built a new server to host vCenter. This had an added bonus of a very easy rollback plan if anything went wrong. All I would have to do is restore the database and boot up the old server. VMWare has an article that outlines part of this process at http://kb.vmware.com/kb/5850444

Installation

To use the existing VirtualCenter database, a system DSN needs to be created to point to the existing database. Enter in the user credentials and set the default database to “VCDB” (or whatever your VirtualCenter database is called) and the rest of the settings should be fine at the defaults. It’s important to note that the VirtualCenter service should be stopped and disabled on the original server before proceeding from here. Bad things can happen if you have two servers trying to access that database.

After creating the DSN, I began installing vCenter4. I was greeted with an error saying: “Please make sure SQL Server Agent service is running on the database server.” This ended up being an issue with some maintenance jobs that I had set up when migrating the database. Renaming them resolved the issue.

The next thing to keep in mind is that VirtualCenter and vCenter make use of SSL certificates. These needed to be copied from the old VirtualCenter server. By default, they are located at %AllUsersProfile%\Application Data\VMWare\VMWare VirtualCenter\SSL. I copied the entire SSL folder onto the new server and vCenter found it without issue.

Post-install Configuration

From here on out, I let the installer do the rest of the work. In about 5 minutes I had a working vCenter server set up. At this point I had to connect the hosts to the new server. Before the ESX hosts can be connected, however, they need to be licensed. vCenter handles this by allowing you to specify the IP address of an external licensing server for legacy hosts. Adding the IP address the old VirtualCenter server will take care of this for the time being.

Each ESX host remembers the IP address of the VirtualCenter server that manages it to prevent conflicts. Right-clicking the server and selecting connect will make vCenter communicate with the hosts and override that information. It will then connect the the new vCenter server to the host.

I had a little trouble with our remote ESX hosts. It’s important that the /tmp/vmware-root directory exists on the target ESX host. If that’s not there, the vCenter server is unable to transfer binaries that are required to set up the connection. The folder can be created by the root user; default permissions should be fine. One of our servers was a little more difficult than the other and still would not connect after creating the temp directory. Restarting the vmware-mgmt service resolved this.

Still to come…

vCenter 4 is now managing our ESX 3 and ESX 3.5 hosts. In the next post I’ll walk through the process of upgrading the host machines and the virtual machines.