Monthly Archives: October 2007

Complete Overkill

This past saturday I spent the whole day at work. Over twelve hours. The day was boring for the most part, tedious, and I even fell asleep once or twice at my desk. Not good you say? The thing is, I wasn’t working. I was migrating data off my personal RAID array so that i could expand it to encompass the second RAID cabinet that I just finished acquiring disks for.

This whole saga goes back to the beginning of the summer. I was in the middle of a pretty large eBay binge, all the while buying tons of compter stuff I didn’t really need. While I certainly didn’t need the stuff, having it around has allowed me to further my knowledge in things like Xen and play with other cool technologies like ATA over Ethernet (AoE). Anyway, during my browsing, I came across a RAID cabinet completely loaded out with 18GB drives pretty much identical to one I had purchased earlier on. The price was pretty cheap, so I nabbed it.

The customary week passed while UPS ground transported the 100 pound-plus package from its origins on the west coast to its new home in Lansing. When it arrvied, I eagerly opened up the box and inventoried its contents, only to find that I wasn’t given all that I was promised. The RAID cabinet itself, the full compliment of twelve 18GB drives, and the mounting rails were all there, but the two external SCSI cables were missing. I promptly got in touch with the seller, and he assured me that it was just an oversight and that he would send the cables right away. Right away turned into a couple of weeks. I finally received the delivery, only to find that he had sent the wrong cables. They would easily fit into the ports on the back of the RAID card in my server, but the other end had the wrong connector type, and wouldn’t connect to the RAID enclosure. I got back in touch with the seller again, and he again promised to right the wrong, and at no charge. Well, those cables never arrived. I was ticked, but I got over it.

Over the next few months I replaced the 18GB disks with 36GB disks so that I could merge the second RAID cabinet into my current RAID array, which is comprised of 36GB disks. Used 36GB drives are relatively inexpensive on eBay so I was able to get the requisite number of disks without too much expense. I also ordered the cables that I was denied in my original purchase.

All the pieces were in place last week, so I decided to try getting everything set up. I got everything put together and connected this past thursday night, and started the process of finalizing things. The RAID card I’m using in the machine (a Compaq/HP Smart Array 5304-128) supports live expansion of an array onto new disks, so I fired up that process from the command line array management utility. I was mildly surprised that I could only extend onto seven of the twelve disks in the second RAID cabinet. I figured that it was just something in the way the expansion algorithm worked, and that I’d be able to expand onto the other disks when the initial expansion finished.

Rearranging 500GB of data on disks is not a quick process, so I had to wait a good twelve hours before I would discover that I was not able to expand the array onto those remaning five disks. I searched around for possible reasons why I was having difficulty and found nothing concrete. I did come across a blurb somewhere that mentioned RAID6 allowing for more disks than RAID5 in an array, which didn’t really seem correct to me, but I decided to try converting the array to RAID6 anyway. Another twelve hours later, I discovered that didn’t work either.

I decided to completely redo the RAID array from scratch, which leads us to last saturday. I was up pretty early, so I made myself a big breakfast and headed into work to do the deed. I got started copying the data off the RAID array, which was no quick process. It takes a while to copy over a hundred gigabytes of data. During this time, I played some games on the laptop, walked around the datacenter, chatted with people, and took a few naps. There’s a surprisingly comfortable position I’ve discovered on my desk where I can just zonk right out…. when I’m not working of course! Anyway, once the data was copied off, I recreated the RAID array and was glad to see that I could use all the disks to their full capacity.

I began the process of copying data back onto the array, and settled in for another nap. I awoke a short time later to find that one of the drives in the array had croaked. It turned out to be one of the disks in the newest group I had purchased, and not really tested all that thoroughly. A disk failure sounds bad, but in RAID6, up to two disks in the array can fail without any data loss. So, in this case, it was just an annoyance. The RAID card detected the failure and immediately started working to rebuild the failed disk’s data on one of the hot spares I have configured. This process slowed down the data transfer pretty substantially, so I was there for a lot longer than I wanted to be. I replaced the failed disk with one of my leftover disks, and the array was happy again.

So, now I have a nice RAID6 array composed of 26 36GB SCSI drives and two hot spares. It weighs in at around 860GB total, and takes up 11U worth of space on the racks at work. I find it pretty funny that there are now single SATA drives that can store more data than that entire array, but they are nowhere near as cool. I have the blinky light factor on my side, as evidenced here…

[RAID setup]

Operation: Eye Health

Here’s a short update for those that check back regularly on the status of my cybernetically-enhanced laserbeam-shooting eye. Well, it’s not really cybernetically enhanced, and it does not yet shoot laser beams, but it is doing remarkably well considering how much its been through. I had another routine checkup with Dr. Saxe last week, and he is still seeing no further evidence of the Coats’ Disease that was plaguing me beforehand. Things have been calm and stable for close to a year now (at least), and I’m still seeing the “slow 20/20” that I’ve been seeing for some time now.

To the others that read up on my situation – may your dealings with Coats’ Disease go as well as mine have.

More Fun With AoE

In another instance of me finding a solution to an imaginary problem, I've succeed in creating a diskless workstation that uses an AoE-exported LVM partition as its root filesystem. I've been wanting to try it out for a while, and this weekend became the time. The setup turned out to be pretty easy once I got past some of the technical hurdles. I didn't do a complete install over AoE this time around, but I've proven it's at least possible to make to make it go using an image made on another machine.

The setup I used is similar to what I've got going with my Xen + AoE setup on my colo boxes at work. I've got a LVM partition that houses the files, and a vblade server exporting that LVM partition over AoE. Out of laziness, I just copied the Fedora7 install from my linux workstation into the LVM partition to provide something I could try booting.

The client side of AoE is provided by a kernel module, so it's not hard to get things working in normal circumstances. Getting it to work on boot was a bit more of a chore, but was pretty easy once I got past the stumbling blocks. I used gPXE/Etherboot on a CD-ROM to get things into a network-bootable state since the onboard network boot stuff on the machine I used is crap and doesn't work. From there, I loaded pxegrub, which then grabbed the kernel and initrd image from my file server via TFTP. I've been booting my MythTV frontend in a similar manner for some time now, only using NFS as the root filesystem.

Getting the initrd right was the biggest hurdle in the whole endeavor. Using mkinitrd I was able to get the AoE and NIC drivers to load without issue. I knew I would have to modify the initrd to include the proper device nodes so that the init script could communicate with the AoE module and see the block devices used to tie into the export, but it wouldn't see the export no matter what I did. Some hours later and after the creation of many initrd files, it occured to me that the network interface was never brought up into an active state. The only way I know to bring up an interface is with ifconfig, so I copied that into the initrd along with the shared libraries it needed. After modifying the init script to issue the command to bring up the interface, everything worked! The machine booted its Fedora7 install right over the network as if the install were on a local hard drive.

As far as usability goes, it's pretty snappy. If you're paying attention, you can see a slight delay in low-latency type things like tab completion at a bash prompt, but loading applications and reading/writing things in bulk is pretty close to on par with a regular local disk.

Now that I've done it, I want to go back and see if it's possible to do a whole install to an AoE export. If I can do the same kernel module tomfoolery before the install process as I did in the initrd, it should be possible to do the install to the AoE share instead of a local disk. Booting would still require PXE, TFTP, and the modified initrd image, but that stuff is easy now that I know how to get things done.

I plan on writing up a more detailed wiki article once I have a bit more motivation, but that time is not now. I didn't have much help from the intarweb in finding this solution, even though there were a few people mentinoning that they'd like to try it. I already get a decent amount of traffic here for my Xen+AoE stuff, so maybe this AoE root stuff will be helpful to others too.

Weekend Update

First things first. The Star Wars Family Guy episode is absolutely freakin hilarious. It is, hands down, the best episode of Family Guy ever. EVER. Moving on…

This weekend has been an exercise in patience. It was the first weekend in a while where I didn’t have some preexisting commitment, so I decided I’d try to get a few projects knocked out. I’ve been meaning to upgrade my Xen server here at home and make use of some of the half-height SCSI drives I mis-ordered a few months back. It turned into a much larger headache than I would have expected – mainly due to my stubbornness. I started the process of setting up the new environment in one of the machines Justin so graciously donated to the Mike Neir Hardware Fund a few months back. It’s a nice server piece with a lot of space for drives, so it worked out well with the four SCSI drives I set up in RAID5. The problem lied in the fact that there were only four power leads coming out of the power supply. That meant no CD-ROM and no easy install. Sure, I could have put another power supply in the machine, but I was lazy yet determined to impose my will upon it, so I decided to make due without replacing the power supply. The machine gave me a good fight though. I ended up using cluster knoppix in terminal server mode to PXE boot the problem machine so I could copy the OS from my current Xen server to said problem machine. That worked, but I had a lot of trouble with the custom Xen kernel I had on the old app server. So, in the interests of laziness and getting everything working, I decided to uprade the whole thing from CentOS 4.4. to CentOS 5, which has a pretty and non-difficult Xen kernel right out of the box. Twelve hours after I started, I prevailed. I probably could have had everything working in less than an hour if I wasn’t lazy and/or stubborn, but doing things the hard way always seems to yield more knowledge than the easy way. I’m a fan of learning and honing my skills, so it’s all good.

After my difficulties with the Xen server, I performed the long-overdue task of cleaning out my fridge. I really need to get better at the task of managing left-overs methinks. There were some items in there that probably dated back to the first Clinton administration. After my cleaning task was complete, I put together a shopping list in the attempt to refill the fridge. I hadn’t been shopping for more than anything but a gallon of milk or mountain dew in close to two months. I made a strong focus on acquiring a lot of raw materials instead of pre-made items since I’ve been experimenting a lot more with actually preparing food instead of just warming it up. I picked up just about every spice I’ve ever heard of, and the receipt reflected it. $276. Yeah. Two hundred and seventy-six dollars. Now I need to put the recipe book I have into action and make some tasty food.

Now I’m unhappy and less motivated to post. The Star Wars episode of Family guy displaced Metalocalypse, which isn’t a good thing. Grumblefest.

Running Through The Woods

Huh. Dreams are weird.

Matt: what was your dream about
Mike: heh
Mike: not rightly sure
Mike: like
Mike: I was out in the woods with a group of people, my mom was part of it
Mike: and we were going to this building
Mike: it apparently was a native american museum of some sort
Mike: we go into this place, and are looking at artifacts
Mike: and we go into this room, and there's this big horizontal wheel-lookin thing
Mike: kinda like the wheel of fortune, but without the colors and money
Matt: weird.
Mike: and the indian folks are doing something on the wheel
Mike: kind of a chant thing
Mike: my mom tells me and the other people in the group to sit and watch it, because it'll probably be educational/interesting/etc
Mike: she goes into another room
Mike: and the indians ask us to volunteer in the ritual they're doing
Mike: we're kinda like "nah, that's ok"
Matt: ok
Mike: then the indians get mad
Mike: about then my mom comes back in the room, and we go over to this cash register along the wall
Mike: and the teller proceeds to charge us $8800 for sitting there and watching the ritual
Mike: and I'm like "wtf $8800?!"
Mike: part of the fee was a $1200 babysitting fee
Mike: so we just up and left... indians were shouting at us and shit
Mike: then, in some seperate timeframe, I'm wandering through the woods
Mike: just kickin around
Mike: and I come across the building again when I'm climbing on some rocks
Mike: and the indians see me
Mike: and they start chasing me
Mike: and then I woke up.
Matt: haha, that is really messed up
Matt: dreams are goofy that way i guess
Matt: lol
Mike: yah seriously.

Rocktober!

Over the past few weeks I’ve been quite the “consumer whore,” as Gene would say. With my recent transition back to bachelor status and a nice raise at work, I’ve had a lot more cash laying around. Because of this, I decided to splurge on some music that’s caught my attention. Over said few weeks, I’ve picked up three albums by Amorphis, two by Meshuggah, and one each by Megadeth, Killswitch Engage, Demiricous, and Iced Earth.

Out of all of the albums I picked up, the Iced Earth album (“Framing Armageddon”) was the only one I had any plans on picking up. It’s the first half of a sci-fi-ish narrative Jon Schaffer has been brewing up for some years now. He laid the foundation of the story quite a while back with the “Something Wicked” trilogy, and things have come to fruition. I was kind of thrown off by the style of the album. It has a lot of the signature Iced Earth elements like Jon Schaffer’s robotic-arm triplet-riffing and a strong vocal presence, but the music has a distinctly different feel in many places. I’m not saying it’s bad, but its not what I expected out of Iced Earth. Since the story is divded into two parts on separate CD releases, it’s my guess that the first CD is a more subdued lead-in, kind of the calm before the storm. We’ll see once the second half is released I guess.

I’m really digging the two Nordic components of my purchases – Amorphis and Meshuggah. The former used to be much more of a doom/deah metal band, but they’ve expanded on the typical musical palette of those genres greatly and incorporated clean vocals and an assortment of typically non-metal instruments (such as saxophones, flutes, and sitars). The result is a very cool blending of styles, and it’s quite pleasure to listen to. Meshuggah, on the other hand, could probably be quite painful to listen to if you weren’t into their particular way of doing things. They’re abrasive, intense, and their compositional style is downright weird. The current last.fm description probably says it better than I ever could.

Among the band's most recognizable qualities are lead guitar player Fredrik Thordendal's abrasive, chaotic and dischordant solos, singer Jens Kidman's vocals, which resemble manic screams and shouts; the churning, dissonant rhythm guitars and the polymetric drum beats. In a typical Meshuggah song, drummer Tomas Haake plays two separate rhythms: a standard 4/4 beat with his hands, and a completely different metrical subdivision with his feet. The guitars mostly follow the bass drum work, creating an awkwardly pulsating rhythmic pattern to work as the basis of the song.To give an example, the main riff of the song "New Millennium Cyanide Christ" from their 1998 album Chaosphere follows the aforementioned blueprint. Haake beats a rather slow 4/4 rhythm with his hands, while the bass drums and guitars play a repetitive 23/16 rhythm pattern on top of it. As the subdivided pattern is repeated, the pattern's accents shift to different beats on each repetition. After repeating the 23/16 pattern five times, a shorter 13/16 pattern is played once. These patterns sum up to 128 16th notes, which equals exactly 8 measures in 4/4 meter.

Speaking of “New Millenium Cyanide Christ” – its music video has to be one of the funniest I’ve ever seen. A band air-guitaring to their own song on their tour bus. Genius idea. Anyway, I digress. Dispite its chaotic nature, I find their stuff quite easy to work to. I found myself getting quite a bit done the other day when I was listening to it at work. Strange.

The remaining three albums have been outshined by the aforementioned works, but I’ll listen to them more eventually. The new Megadeth CD (“United Abominations”) is pretty good. I’ve come to expect their newer stuff to be much more hit-and-miss than their defining works, and this is no different really. There are some pretty fine moments on the CD, and some others that are less stellar. Overall, I’d have to say that its more consistently better than its predecessor (“The System Has Failed”), though. I’m completely in love with Demiricous’ style. A lot of people seem to dog them because they sound a lot like Slayer, but what’s so bad about that? They have certain aspects that sound similar, but overall I think there’s enough differences to keep them from sounding completely derivative. They’ve got a new album coming out in a few weeks, which probably means my trend of CD purchases won’t stop soon. The Killswitch Engage CD I picked up has really been eclipsed by all the other stuff I’ve picked up. I’ll listen to it more, I swear!