Time Warner’s reluctance to issue me a second IP address meant that I’d have to find another solution. SixXS is another tunnel broker which makes use of a protocol called AYIYA. AYIYA is capable of traversing NAT, so it would work well in my situation. Unfortunately it’s not supported by Cisco, so the 2611 was of no use to me anymore.

After some failed experiments and lost money with open source firmware and Linksys routers, I decided that a full Linux install would be the best route to go. I built a machine out of spare parts that I had laying around and installed Ubuntu 9.10 on it. Once I got everything up and running, I installed the SixXS client (AICCU) and started working on getting the tunnel up and running.

I hit another snag when I found out that the point of presence (the tunnel broker’s endpoint) was down. I jumped to freenet6 in the interest of keeping things moving. They provide a tunnel via TSP, which also traverses NAT. Their nearest POP is all the way over in Montreal, which means big latencies for someone in LA, but I was able to establish a tunnel, which I was satisfied with for the moment. Here’s my config for their client (gw6c):

userid=
passwd=
#server=broker.freenet6.net
# Always use Montreal POP
server=montreal.freenet6.net
auth_method=any
prefixlen=64
template=linux
if_tunnel_v6v4=freenet6
if_tunnel_v6udpv4=freenet6
# Assign the v6 address to eth0
if_prefix=eth0
keepalive=yes
keepalive_interval=10
host_type=router

Once I’d established my connection and saw what my v6 IP was going to be, I used a v6 subnet calculator to find out what my /64 networks would be and noted them down. I then entered the v6 address into /etc/network/interfaces so that I could add static routes as the interface came up:

# v6 Interface
auto eth0
iface eth0 inet6 static
    #address 169.254.1.2
    #netmask 255.255.0.0
    address 2001::1
    netmask 64
    up ip -6 route add 2001:0:0:1::/64 via 2001::2 dev eth0

You may notice the link local v4 address that’s been commented out on the v6 address. The reason for this being there is that gw6c can’t do its magic unless the interface specified is up. Until I had my v6 address, I used this link local address. 2001::2, as specified in the route above, is my ASA, which has had it’s address manually configured. 2001:0:0:1::/64 is the network that I’m using for my LAN. I’ve got 254 more networks that I can make use of if I ever find a reason (2001:0:0:2::/64…2001:0:0:ff::/64).

Configuring the ASA was fairly simple. There’s an outside and an inside interface that need to be configured, then the default route needs to be set:

interface Vlan10
 nameif inside
 security-level 100
 ip address 192.168.1.1 255.255.255.0
 ipv6 address 2001:0:0:1::1/64
!
interface Vlan30
 no forward interface Vlan20
 nameif v6tunnel
 security-level 0
 no ip address
 ipv6 address 2001::2/64
!
ipv6 route v6tunnel ::/0 2001::1

As far as I can tell, there’s really no reason I need to use a second VLAN for the v6 tunnel, so I’m going to eventually try moving it to the outside VLAN, but for now it’s working well, and I don’t need a DMZ at the moment.

Once I configured the ASA properly, it started advertising itself and all of the hosts on my LAN are picking up v6 addresses from it. The problem I’m left with is how to do DNS discovery. Unfortunately there doesn’t seem to be a good answer here. Microsoft seems to be fond of the idea of using DHCPv6, Apple wants to use well-known anycast addresses and I honestly have no idea what’s going on in the Linux world.

Upgrading Hosts

VMWare has made it very easy to upgrade hosts to ESX 4. I made use of two remote upgrade options in my migration:

vCenter Update Manager

A centralized update management system which can handle anything from security updates to full ESX upgrades

The Host Update Utility

A standalone application built into the new vSphere client

vCenter Update Manager

For the local host machines, I made use of vCenter’s update manager. This is an add-on for vCenter and it requires its own database, but I highly recommend using it. Its end functionality is similar to any built-in software update mechanism, but its flexibility is what makes it really impressive. For information on installing, please see the administration guide.

Update Manager works on the idea of a “baseline”: a set of minimum requirements that a target needs to meet to be considered compliant. Baselines can be applied at any level in the vCenter hierarchy. To handle the upgrade of our host machines, I created a baseline for ESX hosts specifying that they should be at level 4.0. I then applied this baseline at the datacenter level in both sites. In some cases it may be more apt to apply it at the cluster level, but we have no necessity for legacy hosts in our case.

Once the baseline was applied, I selected the host I wished to update and clicked the update manager tab. I could see, as expected, that it was not compliant with the upgrade baseline applied to it. In a DRS cluster, one should be able to click the remediate button and let vCenter do all of the work to migrate VMs off and put the host in maintenance mode prior to the upgrade. This requires that you keep a pretty clean environment, however.

In our case, we had some hosts that were unable to vMotion either due to snapshots or mapped drives, so I had to take care of things manually. As soon as the host was in maintenance mode, however, choosing remediate rebooted the host and performed the upgrade all on its own within about 20 minutes or so. The progress bar was fairly accurate at most points of the upgrade and once the host was back up, I could see that it was compliant with the baseline.

The Host Update Utility

One major concern in my case was a couple of hosts sitting on the other side of a slow (3mbps) WAN link. I’m sure you’ll agree that most IT tasks are at their best when they go unnoticed. Well, taking up a huge chunk of a site’s bandwidth while I transferred an ISO multiple times would certainly not win me any gold stars.

VMWare’s Update Manger is capable of staging updates on the target for later install, but this only works on hosts that are at 4.0 or above. It would also require moving the ISO over the WAN multiple times, because the updates are staged to the host rather than another vCenter or Update Manager server. This made the Host Update Utility a much more interesting option.

Fortunately, there’s already data replication going on between the main site and the remote one for DR purposes. I was able to mount a snapshot of one of our DR volumes and copy the ISO onto a file server in that site. Launching the host update utility for the first time, I received a prompt to download patches from VMWare. The servers then needed to be added by hostname from the file menu. Once they’re added, the host update utility will do a quick scan to find out what version they’re at. If it’s in an upgradable state, the “Upgrade Host” button will initiate the upgrade.

Overall the procedure from this point on is very similar to the update manager service. The one notable exception is that there is a lot less feedback regarding the status of the host. There were a few times where I was wondering if the hosts were stalled, but eventually the progress bar jumped forward and I could see that things were still working. For this reason, I definitely recommend using the update manager service where possible.

Upgrading VMs

To take full advantage of the new functionality in vSphere 4, the VMs, themselves, must be upgraded. ESX 3.x operated on virtual hardware version 4. vSphere uses version 7, which has a number of notable improvements in storage, networking, and hotswap capability. It’s important to note that there are a few disappointments with the new networking and storage modules, but overall it’s a large step forward. There is a useful article at boche.net that talks about memory and CPU hotplug.

Upgrading the VMs is a two step process requiring at least 2 reboots (my process includes 3). First, the VMWare tools installation must be upgraded in the guest operating system. This may be done either from the vSphere client or from within the guest OS by opening the VMWare tools console. In my case, I chose to upgrade from within the guest so I was keeping an eye on every step of the process. This may not be viable for anyone with a large number of VMs to update. Upgrading the VMWare tools will require a reboot.

Secondly, the virtual hardware needs to be upgraded. The virtual machine needs to be powered down for this step. It’s a simple matter of right-clicking the virtual machine and selecting “Upgrade virtual hardware”. A message will pop up notifying you that this is a one-way process and once it’s finished, your upgraded VM will not work on older versions of ESX. Keep this in mind if you have any legacy hosts staying in your environment.

The next time the guest OS boots, it will detect the new hardware and configure it. For Windows VMs, this will require another reboot. I haven’t, yet, tested the process on a Linux VM. Once the guest OS is finished rebooting, the upgrade process is complete.

Environment

Our environment is fairly small in terms of number of hosts, but we have most of the components in place, which makes for a very nice upgrade experience.

• A VirtualCenter Server v2.5 (primary data center)
• Two host machines v3.5 (primary data center)
• Two host machines v3.0 (disaster recovery data center)

Procedure

The first thing that needs to happen during the migration is upgrading the VirtualCenter server. vCenter 4 is very capable of managing legacy products, but it’s a good idea to take a look at the compatibility matrices just to be safe.

Migration Strategy

The hardware that our current VirtualCenter Server was running on was less than adequate for running vCenter 4. So, I went ahead and built a new server to host vCenter. This had an added bonus of a very easy rollback plan if anything went wrong. All I would have to do is restore the database and boot up the old server. VMWare has an article that outlines part of this process at http://kb.vmware.com/kb/5850444

Installation

To use the existing VirtualCenter database, a system DSN needs to be created to point to the existing database. Enter in the user credentials and set the default database to “VCDB” (or whatever your VirtualCenter database is called) and the rest of the settings should be fine at the defaults. It’s important to note that the VirtualCenter service should be stopped and disabled on the original server before proceeding from here. Bad things can happen if you have two servers trying to access that database.

After creating the DSN, I began installing vCenter4. I was greeted with an error saying: “Please make sure SQL Server Agent service is running on the database server.” This ended up being an issue with some maintenance jobs that I had set up when migrating the database. Renaming them resolved the issue.

The next thing to keep in mind is that VirtualCenter and vCenter make use of SSL certificates. These needed to be copied from the old VirtualCenter server. By default, they are located at %AllUsersProfile%\Application Data\VMWare\VMWare VirtualCenter\SSL. I copied the entire SSL folder onto the new server and vCenter found it without issue.

Post-install Configuration

From here on out, I let the installer do the rest of the work. In about 5 minutes I had a working vCenter server set up. At this point I had to connect the hosts to the new server. Before the ESX hosts can be connected, however, they need to be licensed. vCenter handles this by allowing you to specify the IP address of an external licensing server for legacy hosts. Adding the IP address the old VirtualCenter server will take care of this for the time being.

Each ESX host remembers the IP address of the VirtualCenter server that manages it to prevent conflicts. Right-clicking the server and selecting connect will make vCenter communicate with the hosts and override that information. It will then connect the the new vCenter server to the host.

I had a little trouble with our remote ESX hosts. It’s important that the /tmp/vmware-root directory exists on the target ESX host. If that’s not there, the vCenter server is unable to transfer binaries that are required to set up the connection. The folder can be created by the root user; default permissions should be fine. One of our servers was a little more difficult than the other and still would not connect after creating the temp directory. Restarting the vmware-mgmt service resolved this.

Still to come…

vCenter 4 is now managing our ESX 3 and ESX 3.5 hosts. In the next post I’ll walk through the process of upgrading the host machines and the virtual machines.

A bit of searching on the internet for formatting KB MB, etc. came up with a post by Ajay on http://www.eggheadcafe.com which suggested pasting the following code into the current worksheet’s private module:

Private Sub Worksh5eet_Change(ByVal Target As Range)
    If Not Intersect(Target, Range("C:C")) Is Nothing Then
        If Target.Value < 1000 Then
            Target.NumberFormat = "0 \B"
        ElseIf Target.Value < 999500 Then
            Target.NumberFormat = "0.000, \K\B"
        ElseIf Target.Value < 999500000 Then
            Target.NumberFormat = "0.000,, \M\B"
        ElseIf Target.Value < 999500000000# Then
            Target.NumberFormat = "0.000,,, \G\B"
        Else
            Target.NumberFormat = "0.000,,,, \T\B"
        End If
    End If
End Sub

I had some trouble with the macro and eventually abandoned it, but it did give me some ideas. The actual formats were good and I ended up using them later. The criteria for setting the formatting was spot on as well. After thinking about it for a while, I decided to look into Excel’s conditional formatting feature. There are premade rules for changing the cell color, etc., but it turns out that it’s capable of quite a bit more.

Setting up formatting rules

Conditional formatting is based on sets of rules which are stored in each worksheet. To create a new rule, go to Home > Styles > Conditional Formatting > Manage Rules.... Click New Rule... and select "Format only cells that contain". Select "Cell Value", "less than" and enter "1000" for the value.

Once the condition is set, click Format... set the formatting. In this case we are formatting values that are less than 1KB, so we’ll want to set a custom format of "0 \B"

Make a few more rules for kilobytes, megabytes, etc. You’ll probably want to show a couple of decimal places for the larger numbers. Add a comma for each order of magnitude as you’ll be showing a suffix rather than the extra zeros. You should end up with formats similar to this:

0 \B
0.00, \K\B
0.00,, \M\B
0.00,,, \G\B

Applying the rules

Once you finish creating your rules, you’ll be brought back to the Conditional Formatting Rules Manager. You’ll probably notice that “Current Selection” is selected at the top of the dialog. Change this over to “This Worksheet” for better control. Although it shouldn’t matter too much, I’d recommend ordering the rules from smallest to largest. This should prevent potentially funky behavior if a cell’s value should fall on a boundary between rules. I also suggest leaving “Stop If True” unchecked to let the latter rule win.

You’ll notice that there is an “Applies To” box. This is where we select which cells will be affected by the rules. Click the button and make your selections. Cells in each selection should be adjacent to avoid complications. Use the ctrl key to select multiple ranges. Once you’ve made your selection, click the button again and copy the contents of the box to the other rules and you’re done.

Other thoughts

This approach works based on multiples of 1000 instead of 1024. Setting the criteria to look for the correct boundaries is easy enough by specifying "Cell Value between =Power(2,10) and =Power(2,20)", but I haven’t yet found a way to get Excel to see 1024B as 1KB as opposed to 1.024KB. Any input on this would be quite welcome.

This approach should work well for formatting any units that are base 10; metric units, for instance. The base unit can be chosen arbitrarily, so it’s really quite wide open.

1. Firstly, you should be running service pack 1. I believe the tools are available for the original release of Vista, but I haven’t delved into it.
2. You’ll need to download the updated RSAT pack from Microsoft:

   Microsoft Remote Server Administration Tools for Windows Vista…

   • …for x64-based Systems
   • …for x86-based Systems
3. Go to the control panel and open “Programs and Features”.
4. Click on “Turn Windows features on or off.
5. Find and enable “Remote Server Administration Tools”.

Once you hit OK, your system will spend about 5-10 minutes configuring itself for remote administration and the “Administrative Tools” menu should be populated.

Fortunately, the good people working on the GnuWin32 project have decided that Windows admins deserve a robust command line as well. GnuWin32 is a set of Windows-native executables that duplicate a large amount of the functionality found on a UNIX or Linux system. A lot of these little programs can make a Windows administrator’s life much easier. grep, tail, diff… all available without installing Cygwin.

The packages are distributed in a number of ways. Binaries are packaged in small collections. There’s are packages for the core utilities, diff utilities and separate packages for some of the other utilities like grep. There’s also a setup utility that functions as a package manager to help you keep all of the utilities up to date. Overall, this is a great project that’s definitely worth a look for anyone that hangs out in the CLI much.

]]> http://digital-traffic.net/technology/gnuwin32-a-more-helpful-cli-for-windows/feed/ 0 http://digital-traffic.net/technology/implementing-ipv6-at-home-part-1/ http://digital-traffic.net/technology/implementing-ipv6-at-home-part-1/#comments Fri, 13 Feb 2009 07:33:38 +0000 Brian Shacklett http://digital-traffic.net/?p=40 There have been a lot of articles floating around the net, lately, that mention how quickly we’re running out of IPv4 addresses. Seeing all of this got my interests in IPv6 rekindled. I’d been meaning to get it up and running on my network since I got my Cisco router, but it wasn’t until lately that I was really able to find enough information on the subject to know where to begin. I don’t have anywhere near enough knowledge to write up a tutorial, but I find it helpful and interesting to hear about other peoples’ experience with new technology. I hope it’s helpful for you to see how things went for me.

Selecting a transition mechanism

With regards to deployment, IPv6 is really in its toddler years right now. We’ve moved past the old 6bone test network, but most ISPs still aren’t offering native connectivity yet. That means that the majority of people are going to have to choose a transition mechanism to connect to the rest of the world. There are two main types of transition mechanisms:

• Dual network stacks running IPv4 and IPv6 simultaneously
• Encapsulation of IPv6 packets inside of IPv4 packets (tunneling)

Running dual network stacks is a no-brainer. Even though I’m setting up v6, I still need to talk to all of the v4 hosts that comprise the majority of the current Internet. The real question was what type of tunnel I wanted to use.. There are quite a few different options out there; the most well known being 6to4, Teredo and manual point to point tunneling.

6to4 and Teredo are both automatic tunneling protocols. They don’t require a specific tunnel to be configured and your IPv6 addresses are automatically allocated based on your IPv4 address. Teredo is meant to be used on a single host behind NAT, while 6to4 is generally used on gateway devices. I’d recommend reading the RFCs for more specific information, but a Google search will go a long way as well.

I had a very hard time trying to decide which way to go because I have a dynamic IP address and have no easy way to get a static address. In the end, I chose a manual point to point tunnel. My IP address doesn’t change that often and I decided that the extra control was worth the extra effort of making a change on the tunnel provider’s site once every few months. My tunnel is through Hurricane Electric. They’ve got a very easy setup wizard and even have example configurations for many deployment scenarios.

Requesting and configuring the tunnel

This was the easiest step in the entire setup. I went to http://tunnelbroker.net and signed up for an account. Then I just clicked on “Create Regular Tunnel”. The form asks for your public IP address and gives you a list of servers in different locations to connect to. HE assigns a routed /64 by default, but everyone has the option of allocating a /48 with the click of a link.

Router configuration

After getting the tunnel set up, I went on to configure my router. As I mentioned before, HE has configurations available for the tunnel, so that didn’t take much work at all. The rest I was able to piece together from a Cisco article: Tunneling IPv6 through an IPv4 Network. What I came up with in the end was something like this:

ipv6 unicast-routing
ipv6 cef
!
interface Tunnel0
 description Hurricane Electric IPv6 Tunnel Broker
 no ip address
 ipv6 enable
 ipv6 address 2001:x:x:x::2
 tunnel source x.x.x.x
 tunnel destination x.x.x.x
 tunnel mode ipv6ip
!
interface FastEthernet0/1
 description Inside Network
 ipv6 address 2001:x:x::1/64
 ipv6 enable
!
ipv6 route ::/0 Tunnel0

Line 1 enables unicast IPv6 routing. Line 2 enables Cisco express forwarding for IPv6. The tunnel is configured on lines 4-11. IPv6 is configured and enabled for our inside LAN interface on lines 15 and 16 and I set the default route for IPv6 to the tunnel interface on line 18.

By default, this will issue router advertisements and allow machines to do stateless autoconfiguration for IPv6 addressing. I originally ran into trouble with this while I was trying to subnet my /48. I had my inside LAN interface configured with a /52 and none of the machines on my network would autoconfigure themselves. Eventually I realized that all hosts should be sitting on a /64 subnet. This is in an RFC somewhere. I’ll link to it if I’m able to find it. Once I made the change everything just magically started working.

Final outcome

At this point, I’ve got IPv6 connectivity up and running. I can get to IPv6 enabled sites on the internet by IP address, but I’m not able resolve AAAA records yet because I haven’t got DHCP and DNS configured yet. I’ll get to that in the next post.

The Problem

Dealing with spam that went unrecognized has been more of a manual process. Every once in a while, I’d have to segregate all of my useful mail from the spam and run “sa-learn” on the leftovers. This isn’t horrible, because I tend to shell into my server fairly frequently, but I really prefer to have menial tasks like this automated.

A solution

First of all, I created a folder in my mailbox called “Unrecognized Spam”. The name isn’t important, really. It just needs to be a place to file away all of those messages that SpamAssassin didn’t catch on the way in.
Once that was done, I wrote a very simple little script, which I dropped in /etc/cron.daily/:

#!/bin/bash

SPAM_DIR="/home/bshacklett/Maildir/Unrecognized Spam/cur"

cd "$SPAM_DIR"
sa-learn --spam .;
rm *

Nasty, I know, but it did the job. All I had to do when I got spam that went unnoticed by SpamAssassin was drag it into my “Unrecognized Spam folder” and it would be learned and gone within 24 hours. Of course, I was also getting mail from the cron daemon complaining when there weren’t any emails to learn from or delete.

Improvements

So, this morning I had a little spare time, so I decided to improve on the script a bit:

#!/bin/bash

# Constants
SPAM_PATH="Maildir/.Unrecognized Spam/cur";

# Find all of the directories directly under /home/
homeDirectories=(`find /home/ -maxdepth 1 -mindepth 1 -type d`);

# Loop through the found directories and check for spam
for homeDirectory in ${homeDirectories[*]}
do
    fullSpamPath=${homeDirectory}/${SPAM_PATH};

    #Check if the spam directory exists under this home directory
    if [ -d  "${fullSpamPath}" ]; then

        # Check if there is mail under the spam directory
        if [ "$( ls -A "${fullSpamPath}" )" ]; then
            sa-learn --spam "$fullSpamPath";
            rm "${fullSpammPath}/"*;
        fi
    fi

done

Now I know I’m not a great shell scripter, but this is working pretty well. It basically scans all of the home directories and looks for the “Unrecognized Spam” directory under each one. If it finds it, it will test to make sure that there are emails in the folder, then learn them and remove them.

Caveats

• This isn’t going to scale all that well. I’m guessing it would be fine for 200 users or less, as it runs at night, but it would need some tweaking for anything more.
• As it is, this requires that your mail be stored in the Maildir format. I know that sa-learn can work with mBox stores, but I’m not sure how you’d target it effectively.

Changes

One large change I made to my configuration is using Dovecot’s sasl authentication for Postfix rather than Cyrus. Dovecot really seems to be making headway in the mail server market. I strongly reccomend it to anyone looking for a decent IMAP server.

I’ve also upgraded to WordPress 2.5 and a later version of Roundcube for webmail. Unfortunately, Ubuntu seems to be holding Roundcube back a bit in their package repository. I’m not sure what the holdup is, but they’re at least one version behind at the time I’m writing this. For that matter, WordPress is a couple of releases behind as well. I guess that happens with an LTS release.

Todo

I’m still having a bit of trouble getting WordPress’ permalinks working correctly, so you’ll probably notice that some of the links aren’t working properly on the blog. I remember having this problem in the past, but, for the life of me, I can’t remember what I did to solve it.

Update: Got the permalink issue taken care of. Apache must be told, with the “AllowOverride” parameter, to allow .htaccess files. If this isn’t done, they will be ignored.