Mike Neir's Ramblings

Blinky Lights

Thu, 21 Aug 2008 19:07:24 -0400

It should be the goal of every good geek to build something with lots of blinking lights. I'm proud to say that I've done this many times over. The latest...

Too bad the system this set of drive arrays is attached to is causing me no end of grief.

Ten Times

Tue, 19 Aug 2008 02:04:26 -0400

I'm really tired of needing to apologize to Comcast. They're evil. I know this. But I'm really getting annoyed with having what I perceive to be their problems actually be problems on my end.

I've been having a lot of problems with random cable modem disconnects due to lack of signal over the past few weeks. I know this because I'm a dork and wrote a script that plugs into my Cacti installation and logs the signal levels displayed in my cable modem's admin interface. I did this years ago because - you guessed it - Comcast is evil and their stuff cut out on me enough for me to want to be able to monitor it. There has been a steady and constant trend of ever-dropping signal levels that has made itself apparent since the beginning of the year. I've had the same general layout in the apartment in terms of devices and number signal paths, so I could only assume that the problem was upstream on Comcast's side. I was so confident that this was yet another Comcast screw-up that I ordered DSL.

Well, it wasn't.

I had a service appointment scheduled with Comcast on Friday. The guy did some tests outside where my line plugs in, and he said he noticed a lot of signal "reflection", and said that my splitters and/or wiring were probably bad. He looked around, and also noticed that I was running my cable through the Monster way-too-much-money power strip, and called that a no-no as well. He replaced three splitters with two nicer ones, and ran custom length cables to each one of my devices. The result?

As you can see, the signal levels improved dramatically. The downstream power, which is what I receive from Comcast, is up by at least 10dBmV (depending on the time of day). 10dB corresponds to a ten-fold signal power increase. Signal-to-noise ratio is also better than it was, but not by as striking of an amount.

What's are the lessons here? Use good splitters. Use good cable. Don't assume Comcast is at fault because they're evil.

EDIT: My dad has corrected me. The 10dB increase probably only corresponds to around a 3.1 fold increase in power. I think I read the wikipedia chart wrong.

Xen + AoE + drbd = New Redundant Hotness

Tue, 22 Jul 2008 02:17:02 -0400

A few weeks ago, my buddy Justin posed an interesting problem, one that I've been pondering myself for some time. He's somewhat of a Xen zealot like myself, and is doing some Xen setups that are similar in construction to mine, with a central shared storage array and two or more dom0 machines where the child instances will live. The prospect of migrating domUs between dom0s is quite appealing to him, but he, like myself, realized a critical flaw in the setup. If the storage array fails or requires uptime-affecting maintenance of some sort, the whole setup grinds to a halt. That doesn't really fit the goals he and I are both after.

After a bit of thought, I looked to a project Justin had mentioned a few months back called drbd. It's designers deem it "network raid 1", and that's a pretty accurate description. It's essentially a system that mirrors data between two different machines either in an active-standby or active-active configuration. One of its primary goals is to provide storage as close to 100% of the time as is possible. Its usefulness would vary highly depending on the application. Having a normal file system on it shared between two machines could be nothing short of a nightmare, since neither server knows what the other is doing until changes are already written to the shared storage. A clustered file system would work well with it though. As I began to learn more about how it works, I realized it could potentially be a great solution for my predicament. Since either of of the two machines could provide storage at any given time, it would have no problem fufilling the near 100% uptime requirement.

What really makes the solution stand out to me isn't just drbd itself, but the combination of drbd and AoE. AoE is, by design, a connectionless protocol. When the kernel module is loaded, it does a device discovery to see what devices are available for its use, and listens thereafter for new devices. The information it learns is pretty much limited to a MAC address where the storage device is located and the vblade "addresses" within that device that are available. There's nothing within the protocol that outlaws multiple targets from advertising the same vblade "address", and it's up to the AoE initiator in the kernel module to choose where it's sending data. Because of this, you could have two linux vblade targets running on both "ends" of a drbd setup, and there'd be no conflicts whatsoever. The recommended setup in drbd is to consider a write operation as finished only when data has been written to disk on both "ends" of the drbd setup. Combine that with the fact that AoE will only send commands to one MAC address at a time, and its pretty much guaranteed that both vblade targets will be connected to the same data at all times, even though they're on different machines. I can think of a scenario or two where data would be out of sync, but it would require that disk write operations be done in parallel, and I'm farily certain that they aren't.

The fact that the same data is on both machines and that AoE allows for a quick and painless transfer between vblade targets is what makes this such a simple and effective solution for me. There may be a few seconds of lag while the AoE initiator realizes that the machine it was talking to has disappeared, but that will pass as soon as it does another device discovery and sees the other vblade target. This is perfectly acceptable in my usage scenarios thus far.

I took the plunge a week or so ago and started converting my storage at home to use drbds. It's was pretty simple to convert from my LVM-based setup, since all I had to do was create another single LVM for every partition I wanted to sync between machines. These additional LVM partitions store the metadata that drbd uses to track changes and to keep things in sync. This configuration also allows me to revert back to using "naked" LVM partitions as my vblade storage targets if I decide I don't like drbd in the future. I used my MythTV recording backend as the second drbd server, since it has a lot of space for extra drives and is on pretty much all the time. I put in a 120GB drive, and let everything fly. Once the initial synchronization was complete, I did a few tests, and everything worked as intended. I could kill vblade targets on either machine, and after a few seconds, the initiators would look at the other machine and use it for storage. Success!

As of this weekend I've also converted my setup at work to use the same general configuration. The primary storage consists of a big RAID array, with a secondary machine using a single drive as a backup. I figure that in most cases running in an active-active setup wouldn't be necessary, so I'm going to stick with active-standby, and only start the vblade targets on the secondary machine when I'm planning on a reboot or other maintenance event. I've also considered running in an active-off state (with periodic resyncs), so that there wouldn't be any performance hit from waiting for the second server to complete its writes. This would probably be a less desirable setup since the data could (and very likely would) be out of date if I were to suffer an unplanned outage such as a hardware failure. Nothing I run currently is terribly needy in terms of disk write performance, so I'm not terribly concerned at this point.

Oh Yeah, This Thing.

Sun, 13 Jul 2008 01:18:18 -0400

It's been a pretty crazy couple weeks, for sure. I've been super busy with multiple projects at work, with each one demanding lots of attention. I tend to have a hard time changing directions once I get going on something, so having so many things to do can be pretty stressful. In addition, a system I take care of at work also had some kind of a nervous breakdown, and required a ton of attention in order to correct. A ton of attention while I was on vacation. Totally weak. It's mostly fixed now though, so it's back to being pulled in multiple directions by other stuff. Sigh.

There hasn't been a lot going on outside of work recently due to a general lack of energy and motivation to do anything. All of the stuff happening at work leaves me with no desire to do anything when I get home. It's pretty depressing, in a way.

One relatively fun thing happened though. For the first time since around 2000, I built myself a brand new computer. I got myself a quad-core AMD Phenom processor clocked at 2.3GHz, 4GB of PC2-8500 RAM, a Biostar TPower N750 motherboard, a power supply, and new case from Newegg. I also got a couple of SLI-capable Nvidia GeForce 7600 GS video cards from a coworker. Combine all the parts together and I've got myself a pretty beefy machine. A hell of a lot more beefy than my other desktops. I've been installing some of my old games on it, and they absolutely scream.

Nice One, Stupid

Wed, 11 Jun 2008 00:51:23 -0400

Even though they're an over-priced pseudo-monopoly with a track record of shitty customer service and only somewhat better service uptime, I owe Comcast an apology. My internet was down for most of the past weekend, and for most of Monday as well. I figured it was due to the storms that rolled through on the night of the outage (Saturday), but after a couple of days with no service, I started getting mad. I was cursing their name and anything related to them. I was particularly unhappy when I found out that a coworker that lives in my apartment complex had no interruptions in service. After I heard that, I started thinking of ways that my setup would be sabotaging the process.

And then it hit me. As part of my process to convert my firewall machine into a Xen instance, I altered the physical networking layout so my cable modem would plug directly into my "Core Switch", an old Cisco 2924XL. I gave the cable modem service its own VLAN, which would be accessable via my firewall instance running on a Xen machine. What I failed to consider is that managed switches tend to have features that allow for communication with other switches in order to facilitate ease of management and network health. This communication is typically broadcasted to any device that is listening on regular intervals.

These broadcasts are what caused my issue. In a normal residential cable modem service (with Comcast at least), the cable modem latches on to the first network device it hears traffic from, and assumes that it will be the one it deals with when connecting to the internet. By having my cable modem plugged directly into the switch, it was receiving the switch's broadcast messages before my firewall instance had a chance to make itself heard. Because of this, my firewall's attempts to connect to the internet fell on deaf electronic ears.

This was remedied easily enough by disabling spanning tree protocol on the VLAN that my cable modem connects to, and disabling Cisco Discovery Protocol broadcasts on the port it connects to. I don't like disabling spanning tree, because quite frankly, network loops suck. The chance that somehow make a loop in that VLAN is pretty damn low though, so there's not much to worry about.

Let this be a lesson to those with way too much time on their hands, like myself.

Grayscale

Mon, 19 May 2008 00:03:33 -0400

I had the camera out the other night, and snapped some pictures from the balcony. They turned out pretty cool I think.

Dream Theater, Opeth, Between The Buried and Me, 3 - 5/14/2008

Sat, 17 May 2008 21:46:50 -0400

This was a show that I had greatly anticipated for a long time. Opeth and Dream Theater are easily two of my favorite bands, so when I saw them both on the same bill many months ago, I was pretty excited. Both bands put on great live shows as well, so I knew I wouldn't be disappointed. I made the trek down to Detroit with my dad, Jessica, and Mike.

The first band to play was 3. I thought that we'd get there with plenty of time to see the opening bands, but we didn't. We only caught the last part of one of their songs. It sounded pretty good though, from the five minutes or so that we heard. They had a two drummer setup going, and they were seriously beating on the skins. I would have liked to see more of their set, but alas, it wasn't meant to be. Grade: N/A

Between the Buried and Me played second. I hadn't really been exposed to them in great depth, and for some reason I had the impression that they were going to be another dripping-with-testosterone type band (like Throwdown or Bury Your Dead), but I was pleasantly surprised that they weren't. They do have the crappy barking vocals that is typical for a hardcore/metalcore band, but their music is way more complex. These guys can flat out play, and they pull a ton of different musical styles into their sound. It makes for a pretty cool experience. They only had a half hour so on stage, and they played all of two distinct songs. They wasted no time with blathering between songs either... they pretty much stopped only to announce the name of the second song and get a drink of water, then started playing again. They definitely merited further attention, which is why I bought three of their CDs yesterday. Grade: A

Opeth is the band that I looked forward to seeing the most out of the four. I've only seen them once before (compared to Jessica's ten[!]), and that was an abbreviated set at last year's Gigantour, where they were only able to play four songs. They ended up getting around an hour on stage this time around, and were able to play six songs - Master's Apprentices, In My Time of Need, Baying of the Hounds, Heir Apparent, Wreath, and The Drapery Falls. They had a pretty good sampling of songs from their newer albums, but none from the older four albums, which kind of disappoints me. "Still Life" and "My Arms, Your Hearse" are my favorite albums of theirs, and with two shows under my belt, I've yet to see them play any songs from those albums live. Oh well... there's always next time! Mikael Akerfeldt does a pretty good job of making the breaks between songs interesting as well. He's always got something funny to say, and it's pretty obvious to me that he genuinely enjoys putting on a show for the crowd. Grade: A

Dream Theater closed out the show in the way that only they can - with authority. This was the fourth time that my dad and I saw Dream Theater together, and they've never once disappointed. I've blathered on and on about how good they are in previous posts about them, so I'll try to save it this time around. This was the first time, however, that I saw them without having heard their most recent release. Their previous release ("Octavarium"), seemed kind of tired to me, and didn't really grab my interest much at all. Because of that, I didn't run out and grab their newest ("Systematic Chaos") when it came out last year. I probably should have grabbed it sooner. They played a few tunes from it, and they were pretty good. They opened the set with an instrumental passage from one of those songs, and I was really digging it. Their set list was expansive, and pretty typical for the shows I've seen. They played a good sampling from their albums, and played most of the first act from "Scenes from a Memory". They didn't play "Home," which kind of pissed me off since it is one of my favorite songs of theirs, but hey, you can't win 'em all.

One neat thing they did for this tour was turning something that is typically incredibly lame into something totally awesome. Jordan Rudess (the keyboardist) played a key-tar. Yep, the dreaded 80's pop instrument. In this case though, it was sweet because it gave him the opportunity to step out from behind is mammoth keyboard rig and get up close and personal with the fans like the other band members get to do. He mostly used it in extended "improvised" musical passages that break out of their normal song structures, and used his mobility to add a more visual flare to the frequent traded/harmonized leads that he does with John Petrucci in so many of their songs. I laughed when I first saw it, but it really did add a pretty cool element to their performance, which was already top notch.

The only downer in the show was that they played one of their new songs (I think - I didn't recognize it) for far too long, and it totally slowed their momentum. It was a slow piece of music, and it seemed to go on FOR EVER. Otherwise, they were phenomenal as always. Grade: A

Hot Damn.

Fri, 09 May 2008 00:01:23 -0400

I've talked about Meshuggah once or twice here, and it's pretty safe to say that I'm a fan. Their sound is abrasive - scratch that - caustic to most ears, but I love it. There's so much musical skill in that band that's its not even funny. They've laid siege to my last.fm profile over the past few months, and for good reason. They rule.

So does this kid. Apparently the band held some sort of competition to see who could play one of their new songs ("Bleed") the best on drums, and this 14 year old kid won.

I've been playing guitar for more than 2/3 of this kid's life, and I've achieved nowhere near the mastery this kid displays. Congrats Zach Schiermann, you rule.

Addendum: For whatever reason, I feel like making mention of this. For the song "Rational Gaze," which is one of my favorite 'of the moment' songs, Meshuggah plays on 8 string gutars, tuned to F. Yes, F. One half-step short of a full octave below a normal guitar tuning. Each guitarist is practically playing a bass. Insanity. Delicious auditory insanity.

Crisis Averted

Thu, 08 May 2008 11:13:13 -0400

The situation with my apartment complex has been resolved. Apparently they misplaced the payment records for around 30 residents, so i wasn't the only one that came to them completely pissed off. Everything is kosher again, but I took my opportunity to tell the rep I was dealing with how dissatisfied I was with the customer service in their organization. I still can't understand how they wouldn't attempt to make a phone call before initiating legal procedures to kick someone out of their home. Perhaps that's a reason I'm not in the apartment renting business.

So. Pissed. Off.

Tue, 06 May 2008 20:43:50 -0400

They did it to me again.

I came home from work today in a pretty decent mood. I was jamming out to some good old viking metal, with little on my mind but conquering ancient lands using nothing but sword and shield. Okay, I really wasn't thinking about conquering ancient lands, but it fit. Sue me. Anyway, I get home and check my mail, since I'm expecting my Dream Theater/Opeth tickets to show up soon. No tickets, but I did find a letter from the apartment complex.

They fucked up my payment again, and once again sent me a legal-looking document stating that I owe them money. The problem is that I don't. I checked my bank statement, and they debited my account on 5/1, the day it was due. I've got a confirmation from their online billing system stating that I did indeed schedule payment for that date.

I really don't understand how they could screw that up. My account with them is directly tied to my apartment. When I go to make a payment, the interface has my address and unit number right there at the top next to my name. I started making payments this way because I figured it would be harder for them to screw up. Apparently I have underestimated them.

To make things more aggravating, I couldn't go deal with it today. I got home and opened the letter at about 7:05, and their office closed at 7:00. I went up to their office anyway to see if anyone was lingering, but no. At 7:12, the place was completely deserted. Its probably good for them though. After listening to that viking metal, I was ready to do battle - verbally at least.