Tuesday, July 22 2008, 2:17 AM

A few weeks ago, my buddy Justin posed an interesting problem, one that I've been pondering myself for some time. He's somewhat of a Xen zealot like myself, and is doing some Xen setups that are similar in construction to mine, with a central shared storage array and two or more dom0 machines where the child instances will live. The prospect of migrating domUs between dom0s is quite appealing to him, but he, like myself, realized a critical flaw in the setup. If the storage array fails or requires uptime-affecting maintenance of some sort, the whole setup grinds to a halt. That doesn't really fit the goals he and I are both after.

After a bit of thought, I looked to a project Justin had mentioned a few months back called drbd. It's designers deem it "network raid 1", and that's a pretty accurate description. It's essentially a system that mirrors data between two different machines either in an active-standby or active-active configuration. One of its primary goals is to provide storage as close to 100% of the time as is possible. Its usefulness would vary highly depending on the application. Having a normal file system on it shared between two machines could be nothing short of a nightmare, since neither server knows what the other is doing until changes are already written to the shared storage. A clustered file system would work well with it though. As I began to learn more about how it works, I realized it could potentially be a great solution for my predicament. Since either of of the two machines could provide storage at any given time, it would have no problem fufilling the near 100% uptime requirement.

What really makes the solution stand out to me isn't just drbd itself, but the combination of drbd and AoE. AoE is, by design, a connectionless protocol. When the kernel module is loaded, it does a device discovery to see what devices are available for its use, and listens thereafter for new devices. The information it learns is pretty much limited to a MAC address where the storage device is located and the vblade "addresses" within that device that are available. There's nothing within the protocol that outlaws multiple targets from advertising the same vblade "address", and it's up to the AoE initiator in the kernel module to choose where it's sending data. Because of this, you could have two linux vblade targets running on both "ends" of a drbd setup, and there'd be no conflicts whatsoever. The recommended setup in drbd is to consider a write operation as finished only when data has been written to disk on both "ends" of the drbd setup. Combine that with the fact that AoE will only send commands to one MAC address at a time, and its pretty much guaranteed that both vblade targets will be connected to the same data at all times, even though they're on different machines. I can think of a scenario or two where data would be out of sync, but it would require that disk write operations be done in parallel, and I'm farily certain that they aren't.

The fact that the same data is on both machines and that AoE allows for a quick and painless transfer between vblade targets is what makes this such a simple and effective solution for me. There may be a few seconds of lag while the AoE initiator realizes that the machine it was talking to has disappeared, but that will pass as soon as it does another device discovery and sees the other vblade target. This is perfectly acceptable in my usage scenarios thus far.

I took the plunge a week or so ago and started converting my storage at home to use drbds. It's was pretty simple to convert from my LVM-based setup, since all I had to do was create another single LVM for every partition I wanted to sync between machines. These additional LVM partitions store the metadata that drbd uses to track changes and to keep things in sync. This configuration also allows me to revert back to using "naked" LVM partitions as my vblade storage targets if I decide I don't like drbd in the future. I used my MythTV recording backend as the second drbd server, since it has a lot of space for extra drives and is on pretty much all the time. I put in a 120GB drive, and let everything fly. Once the initial synchronization was complete, I did a few tests, and everything worked as intended. I could kill vblade targets on either machine, and after a few seconds, the initiators would look at the other machine and use it for storage. Success!

As of this weekend I've also converted my setup at work to use the same general configuration. The primary storage consists of a big RAID array, with a secondary machine using a single drive as a backup. I figure that in most cases running in an active-active setup wouldn't be necessary, so I'm going to stick with active-standby, and only start the vblade targets on the secondary machine when I'm planning on a reboot or other maintenance event. I've also considered running in an active-off state (with periodic resyncs), so that there wouldn't be any performance hit from waiting for the second server to complete its writes. This would probably be a less desirable setup since the data could (and very likely would) be out of date if I were to suffer an unplanned outage such as a hardware failure. Nothing I run currently is terribly needy in terms of disk write performance, so I'm not terribly concerned at this point.

aoe drbd linux lvm mythtv raid xen

Sunday, July 13 2008, 1:18 AM

It's been a pretty crazy couple weeks, for sure. I've been super busy with multiple projects at work, with each one demanding lots of attention. I tend to have a hard time changing directions once I get going on something, so having so many things to do can be pretty stressful. In addition, a system I take care of at work also had some kind of a nervous breakdown, and required a ton of attention in order to correct. A ton of attention while I was on vacation. Totally weak. It's mostly fixed now though, so it's back to being pulled in multiple directions by other stuff. Sigh.

There hasn't been a lot going on outside of work recently due to a general lack of energy and motivation to do anything. All of the stuff happening at work leaves me with no desire to do anything when I get home. It's pretty depressing, in a way.

One relatively fun thing happened though. For the first time since around 2000, I built myself a brand new computer. I got myself a quad-core AMD Phenom processor clocked at 2.3GHz, 4GB of PC2-8500 RAM, a Biostar TPower N750 motherboard, a power supply, and new case from Newegg. I also got a couple of SLI-capable Nvidia GeForce 7600 GS video cards from a coworker. Combine all the parts together and I've got myself a pretty beefy machine. A hell of a lot more beefy than my other desktops. I've been installing some of my old games on it, and they absolutely scream.

amd biostar frustrations nvidia work

Wednesday, June 11 2008, 12:51 AM

Even though they're an over-priced pseudo-monopoly with a track record of shitty customer service and only somewhat better service uptime, I owe Comcast an apology. My internet was down for most of the past weekend, and for most of Monday as well. I figured it was due to the storms that rolled through on the night of the outage (Saturday), but after a couple of days with no service, I started getting mad. I was cursing their name and anything related to them. I was particularly unhappy when I found out that a coworker that lives in my apartment complex had no interruptions in service. After I heard that, I started thinking of ways that my setup would be sabotaging the process.

And then it hit me. As part of my process to convert my firewall machine into a Xen instance, I altered the physical networking layout so my cable modem would plug directly into my "Core Switch", an old Cisco 2924XL. I gave the cable modem service its own VLAN, which would be accessable via my firewall instance running on a Xen machine. What I failed to consider is that managed switches tend to have features that allow for communication with other switches in order to facilitate ease of management and network health. This communication is typically broadcasted to any device that is listening on regular intervals.

These broadcasts are what caused my issue. In a normal residential cable modem service (with Comcast at least), the cable modem latches on to the first network device it hears traffic from, and assumes that it will be the one it deals with when connecting to the internet. By having my cable modem plugged directly into the switch, it was receiving the switch's broadcast messages before my firewall instance had a chance to make itself heard. Because of this, my firewall's attempts to connect to the internet fell on deaf electronic ears.

This was remedied easily enough by disabling spanning tree protocol on the VLAN that my cable modem connects to, and disabling Cisco Discovery Protocol broadcasts on the port it connects to. I don't like disabling spanning tree, because quite frankly, network loops suck. The chance that somehow make a loop in that VLAN is pretty damn low though, so there's not much to worry about.

Let this be a lesson to those with way too much time on their hands, like myself.

cisco comcast frustrations linux networking stp vlan xen

Monday, May 19 2008, 12:03 AM

I had the camera out the other night, and snapped some pictures from the balcony. They turned out pretty cool I think.

photography

Saturday, May 17 2008, 9:46 PM

This was a show that I had greatly anticipated for a long time. Opeth and Dream Theater are easily two of my favorite bands, so when I saw them both on the same bill many months ago, I was pretty excited. Both bands put on great live shows as well, so I knew I wouldn't be disappointed. I made the trek down to Detroit with my dad, Jessica, and Mike.

The first band to play was 3. I thought that we'd get there with plenty of time to see the opening bands, but we didn't. We only caught the last part of one of their songs. It sounded pretty good though, from the five minutes or so that we heard. They had a two drummer setup going, and they were seriously beating on the skins. I would have liked to see more of their set, but alas, it wasn't meant to be. Grade: N/A

Between the Buried and Me played second. I hadn't really been exposed to them in great depth, and for some reason I had the impression that they were going to be another dripping-with-testosterone type band (like Throwdown or Bury Your Dead), but I was pleasantly surprised that they weren't. They do have the crappy barking vocals that is typical for a hardcore/metalcore band, but their music is way more complex. These guys can flat out play, and they pull a ton of different musical styles into their sound. It makes for a pretty cool experience. They only had a half hour so on stage, and they played all of two distinct songs. They wasted no time with blathering between songs either... they pretty much stopped only to announce the name of the second song and get a drink of water, then started playing again. They definitely merited further attention, which is why I bought three of their CDs yesterday. Grade: A

Opeth is the band that I looked forward to seeing the most out of the four. I've only seen them once before (compared to Jessica's ten[!]), and that was an abbreviated set at last year's Gigantour, where they were only able to play four songs. They ended up getting around an hour on stage this time around, and were able to play six songs - Master's Apprentices, In My Time of Need, Baying of the Hounds, Heir Apparent, Wreath, and The Drapery Falls. They had a pretty good sampling of songs from their newer albums, but none from the older four albums, which kind of disappoints me. "Still Life" and "My Arms, Your Hearse" are my favorite albums of theirs, and with two shows under my belt, I've yet to see them play any songs from those albums live. Oh well... there's always next time! Mikael Akerfeldt does a pretty good job of making the breaks between songs interesting as well. He's always got something funny to say, and it's pretty obvious to me that he genuinely enjoys putting on a show for the crowd. Grade: A

Dream Theater closed out the show in the way that only they can - with authority. This was the fourth time that my dad and I saw Dream Theater together, and they've never once disappointed. I've blathered on and on about how good they are in previous posts about them, so I'll try to save it this time around. This was the first time, however, that I saw them without having heard their most recent release. Their previous release ("Octavarium"), seemed kind of tired to me, and didn't really grab my interest much at all. Because of that, I didn't run out and grab their newest ("Systematic Chaos") when it came out last year. I probably should have grabbed it sooner. They played a few tunes from it, and they were pretty good. They opened the set with an instrumental passage from one of those songs, and I was really digging it. Their set list was expansive, and pretty typical for the shows I've seen. They played a good sampling from their albums, and played most of the first act from "Scenes from a Memory". They didn't play "Home," which kind of pissed me off since it is one of my favorite songs of theirs, but hey, you can't win 'em all.

One neat thing they did for this tour was turning something that is typically incredibly lame into something totally awesome. Jordan Rudess (the keyboardist) played a key-tar. Yep, the dreaded 80's pop instrument. In this case though, it was sweet because it gave him the opportunity to step out from behind is mammoth keyboard rig and get up close and personal with the fans like the other band members get to do. He mostly used it in extended "improvised" musical passages that break out of their normal song structures, and used his mobility to add a more visual flare to the frequent traded/harmonized leads that he does with John Petrucci in so many of their songs. I laughed when I first saw it, but it really did add a pretty cool element to their performance, which was already top notch.

The only downer in the show was that they played one of their new songs (I think - I didn't recognize it) for far too long, and it totally slowed their momentum. It was a slow piece of music, and it seemed to go on FOR EVER. Otherwise, they were phenomenal as always. Grade: A

btbam concerts dreamtheater music opeth reviews

Friday, May 09 2008, 12:01 AM

I've talked about Meshuggah once or twice here, and it's pretty safe to say that I'm a fan. Their sound is abrasive - scratch that - caustic to most ears, but I love it. There's so much musical skill in that band that's its not even funny. They've laid siege to my last.fm profile over the past few months, and for good reason. They rule.

So does this kid. Apparently the band held some sort of competition to see who could play one of their new songs ("Bleed") the best on drums, and this 14 year old kid won.

I've been playing guitar for more than 2/3 of this kid's life, and I've achieved nowhere near the mastery this kid displays. Congrats Zach Schiermann, you rule.

Addendum: For whatever reason, I feel like making mention of this. For the song "Rational Gaze," which is one of my favorite 'of the moment' songs, Meshuggah plays on 8 string gutars, tuned to F. Yes, F. One half-step short of a full octave below a normal guitar tuning. Each guitarist is practically playing a bass. Insanity. Delicious auditory insanity.

guitar meshuggah music

Thursday, May 08 2008, 11:13 AM

The situation with my apartment complex has been resolved. Apparently they misplaced the payment records for around 30 residents, so i wasn't the only one that came to them completely pissed off. Everything is kosher again, but I took my opportunity to tell the rep I was dealing with how dissatisfied I was with the customer service in their organization. I still can't understand how they wouldn't attempt to make a phone call before initiating legal procedures to kick someone out of their home. Perhaps that's a reason I'm not in the apartment renting business.

apartment frustrations

Tuesday, May 06 2008, 8:43 PM

They did it to me again.

I came home from work today in a pretty decent mood. I was jamming out to some good old viking metal, with little on my mind but conquering ancient lands using nothing but sword and shield. Okay, I really wasn't thinking about conquering ancient lands, but it fit. Sue me. Anyway, I get home and check my mail, since I'm expecting my Dream Theater/Opeth tickets to show up soon. No tickets, but I did find a letter from the apartment complex.

They fucked up my payment again, and once again sent me a legal-looking document stating that I owe them money. The problem is that I don't. I checked my bank statement, and they debited my account on 5/1, the day it was due. I've got a confirmation from their online billing system stating that I did indeed schedule payment for that date.

I really don't understand how they could screw that up. My account with them is directly tied to my apartment. When I go to make a payment, the interface has my address and unit number right there at the top next to my name. I started making payments this way because I figured it would be harder for them to screw up. Apparently I have underestimated them.

To make things more aggravating, I couldn't go deal with it today. I got home and opened the letter at about 7:05, and their office closed at 7:00. I went up to their office anyway to see if anyone was lingering, but no. At 7:12, the place was completely deserted. Its probably good for them though. After listening to that viking metal, I was ready to do battle - verbally at least.

apartment frustrations

Saturday, May 03 2008, 12:38 AM

Everything makes sense now.

humor

Friday, May 02 2008, 11:15 PM

Touching on a few things from my last post...

Moving the bed to the other bedroom seems to have helped. I've slept pretty well the past few days. This is a good thing.

I was finally able to get the firmware on my RAID card updated. It only took about 5 hours worth of time, 5 wasted CD-ROMs, and a lot of annoyance. The result? The RAID array rebuilt itself for two days (pretty damn long for an array of that size), and the card promptly kernel panicked upon reaching 100% on the rebuild process. At least it didn't start over again like it had in the past.

I think I might need professional help. This morning, I built a small cardboard baffle (?) to direct air out of a HVAC duct towards the servers in my closet. People more sane than myself would probably say things like "you should have fewer computers", "why do you have so many computers?", or even the dreaded "you're freakin' weird!" I, however, build cardboard ducting.

I also caught this guy outside my window the other morning. I'm surprised the picture came out so clear... the window I took the picture through is dirty as hell.

frustrations hp photography raid