More Fun With AoE

In another instance of me finding a solution to an imaginary problem, I've succeed in creating a diskless workstation that uses an AoE-exported LVM partition as its root filesystem. I've been wanting to try it out for a while, and this weekend became the time. The setup turned out to be pretty easy once I got past some of the technical hurdles. I didn't do a complete install over AoE this time around, but I've proven it's at least possible to make to make it go using an image made on another machine.

The setup I used is similar to what I've got going with my Xen + AoE setup on my colo boxes at work. I've got a LVM partition that houses the files, and a vblade server exporting that LVM partition over AoE. Out of laziness, I just copied the Fedora7 install from my linux workstation into the LVM partition to provide something I could try booting.

The client side of AoE is provided by a kernel module, so it's not hard to get things working in normal circumstances. Getting it to work on boot was a bit more of a chore, but was pretty easy once I got past the stumbling blocks. I used gPXE/Etherboot on a CD-ROM to get things into a network-bootable state since the onboard network boot stuff on the machine I used is crap and doesn't work. From there, I loaded pxegrub, which then grabbed the kernel and initrd image from my file server via TFTP. I've been booting my MythTV frontend in a similar manner for some time now, only using NFS as the root filesystem.

Getting the initrd right was the biggest hurdle in the whole endeavor. Using mkinitrd I was able to get the AoE and NIC drivers to load without issue. I knew I would have to modify the initrd to include the proper device nodes so that the init script could communicate with the AoE module and see the block devices used to tie into the export, but it wouldn't see the export no matter what I did. Some hours later and after the creation of many initrd files, it occured to me that the network interface was never brought up into an active state. The only way I know to bring up an interface is with ifconfig, so I copied that into the initrd along with the shared libraries it needed. After modifying the init script to issue the command to bring up the interface, everything worked! The machine booted its Fedora7 install right over the network as if the install were on a local hard drive.

As far as usability goes, it's pretty snappy. If you're paying attention, you can see a slight delay in low-latency type things like tab completion at a bash prompt, but loading applications and reading/writing things in bulk is pretty close to on par with a regular local disk.

Now that I've done it, I want to go back and see if it's possible to do a whole install to an AoE export. If I can do the same kernel module tomfoolery before the install process as I did in the initrd, it should be possible to do the install to the AoE share instead of a local disk. Booting would still require PXE, TFTP, and the modified initrd image, but that stuff is easy now that I know how to get things done.

I plan on writing up a more detailed wiki article once I have a bit more motivation, but that time is not now. I didn't have much help from the intarweb in finding this solution, even though there were a few people mentinoning that they'd like to try it. I already get a decent amount of traffic here for my Xen+AoE stuff, so maybe this AoE root stuff will be helpful to others too.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>