Hot Migration Action :: Mike Neir's Page

As described in my last few posts, I've recently acquired a good amount of new server hardware. Well, everything is in my posession now except a few sticks of RAM, and it's all set up at work. I ended up picking up the RAID enclosure I mentioned earlier, along with disks to fill it. It ended up being quite a bargain, with the enclosure, drive trays, and a external SCSI cable only costing around $50 plus shipping. Here's all the new gear mounted in a rack at work... My stuff is the white stuff in a sea of black servers.

I've got the RAID enclosure connected to the dual P3 1.0GHz machine I bought (furthest away on the bottom), and combined, there's 18 drive bays available to the SCSI system. I've got fifteen 36GB drives (plus one hot spare) in a RAID5 storage array and two 18GB drives in RAID1 for the OS installation. The RAID5 array weighs in at about 500GB, so I have plenty of room to keep stuff that I don't want to lose.

I'm currently seeing how well my Xen domains function with NFS root filesystems. So far it looks pretty good. I've got the domains that host my web site (among other things) and the mysql domain running off the RAID5 array via NFS, and I haven't noticed any slowdowns whatsoever. The only unexpected thing I've come across is a few weird incompatibilities with the Gentoo init scripts, specifically when it tries to bring up networking devices. It just hangs up when trying to initialize eth1, which is the interface that the NFS root filesystem is accessed through. My firewall script also kills things, but I should be able to fix that.

Having things running over NFS allows for live migration of running domains. I tried it out a few hours ago, and it's surprisingly painless, given that the appropriate functionality is enabled in the Xen daemon. One command sends a running domain between physical Xen hosts, which is pretty damned neat. I can see this being tremendously useful in a high-availabilty sort of environment. If a host machine needs maintenance, you can simply transfer the running child domain to another host, do your business, and transfer it back with only a fraction of a second of downtime.

Gregg :: 05/17/2007, 12:14 am :: Reply

gotta admit.. its over kill.. but still hot..

sneezer :: 05/17/2007, 2:11 pm :: Reply

i have the same set up but it has a canorgly interfaced with the Xen AND it's stacked with a NFS - also my 6 domains are not enough so I have doubled the thru-put and capped it with the root server. Now it is 6.8! Unreal turnarounds and increased run times! Now that's migration!

Phil :: 05/25/2007, 4:41 pm :: Reply

We lost a RAID 5 array at work a while back. Two drives went down at the same time. We had to pay the big bucks to get the data off. So, don't think that your RAID is bullet proof.

Mike Neir :: 05/25/2007, 9:22 pm :: Reply

It's true that they're not bullet proof, but I've got a hot-spare in the array, so a failure should be corrected automatically.

Michael :: 05/31/2007, 11:11 pm :: Reply

NFS isn't bad if you're not doing heavy I/O. You could also export your partitions as iSCSI targets and mount them that way.