Tales Of The Voting Booth

The American election system is in trouble. Why you ask? The strings holding the pens to the voting booths are attached at the right side, and are much too short! How is a lefty supposed to vote if he can’t mark the right areas on the ballot!?

I kid, I kid.

A funny/odd coincidence happened to me after I filled out my ballot. I placed the ballot in the scanner machine, and it registered the number “666”, which I assume means that I was the 666th person to vote. Some of the older ladies running the voting area were like “uh oh! 6-6-6!” After I got back to the car, I turned on my iPod, which was on random. The Slayer song “Cult” came on as I was buckling my seat belt, which, oddly enough, has the following lyrics..

I've made my choice. Six six six

While I doubt they were referring to the voting process, the coincidence struck me as funny. I guess that means I’ll be voting Slayer in 2012.


As most people probably can discern by reading the backlog of content entries here, I’m a fan of Memcache. It makes database driven applications crazy fast when used properly, and can be damn simple to integrate if your code is set up properly. I integrated it with the monitoring system I wrote at work a while back to take some of the load off MySQL, and it’s worked exceptionally well.

While checking something else out in the monitoring system, I ran a report on the current Memcache status, and was kind of blown away. The system currently consists of 16 servers, 15 of which have memcached running on them, with 512MB allocated on each. That’s 7.5GB of cache memory, for those that don’t like doing math. I’ll let the report speak for itself…
uptime: 563.945474537 days
read: 18097.7607219 GB
written: 19537.6376228 GB192.168.0.3:11211
uptime: 563.944502315 days
read: 18373.8532251 GB
written: 19785.8979402 GB
uptime: 429.195578704 days
read: 13156.7254652 GB
written: 14532.1054876 GB
uptime: 563.944247685 days
read: 19168.9044306 GB
written: 20775.6700451 GB
uptime: 563.944212963 days
read: 18374.297407 GB
written: 19933.5923319 GB
uptime: 563.944178241 days
read: 17361.7501419 GB
written: 18974.0357142 GB
uptime: 563.942581019 days
read: 17525.4281332 GB
written: 18935.3812126 GB
uptime: 317.73099537 days
read: 11330.5213323 GB
written: 12179.3746607 GB
uptime: 0.774074074074 days
read: 3.608856056 GB
written: 5.957609009 GB
uptime: 270.114976852 days
read: 9460.02688244 GB
written: 10477.9305441 GB
uptime: 446.070590278 days
read: 14058.0880373 GB
written: 15349.2392257 GB
uptime: 433.269293981 days
read: 13117.9202484 GB
written: 14265.9189342 GB
uptime: 288.806412037 days
read: 9547.86924515 GB
written: 10396.3484227 GB
uptime: 98.0899074074 days
read: 3658.49374042 GB
written: 4026.71671353 GB
uptime: 18.6412731481 days
read: 529.824877851 GB
written: 591.686248784 GB

uptime (total): 5686.35829861 days
bytes read (total): 183.765072745 TB
bytes written (total): 199.767492713 TB

In just the current running processes, there is over 15 years of combined uptime, with 9 memcached processes up for over a year… 183 terabytes of data read from the memcached processes and 199 terabytes written. That’s a lot of uptime and a lot of data! I’ve got a couple of crash-happy servers in the cluster too. Without those, the numbers would be higher.

For those wondering, the write number is higher than the read number because the latest snapshot data is stored into memcache for easy retrieval each time data is collected. The read numbers are part of the snapshot storage process as well, and would also be higher if there was more activity on the web interface.


This is great too!


It has been a loooong time since I posted. Yes, I’m still alive (obviously). Hopefully posting funny malware emails will make me more chatty.

From: “Microsoft Corp.”
Subject: Security Update for OS Microsoft Windows

Dear Microsoft Customer,

Please notice that Microsoft company has recently issued a Security Update for OS Microsoft Windows. The update applies to the following OS versions: Microsoft Windows 98, Microsoft Windows 2000, Microsoft Windows Millenium, Microsoft Windows XP, Microsoft Windows Vista.

Please notice, that present update applies to high-priority updates category. In order to help protect your computer against security threats and performance problems, we strongly recommend you to install this update.

Since public distribution of this Update through the official website http://www.microsoft.com would have result in efficient creation of a malicious software, we made a decision to issue an experimental private version of an update for all Microsoft Windows OS users.

As your computer is set to receive notifications when new updates are available, you have received this notice.

In order to start the update, please follow the step-by-step instruction:
1. Run the file, that you have received along with this message.
2. Carefully follow all the instructions you see on the screen.

If nothing changes after you have run the file, probably in the settings of your OS you have an indication to run all the updates at a background routine. In that case, at this point the upgrade of your OS will be finished.

We apologize for any inconvenience this back order may be causing you.

Thank you,

Steve Lipner
Director of Security Assurance
Microsoft Corp.

Version: PGP 7.1


Attached was the file “KB779778.exe”. Yah… I’m totally going to run that! I think my favorite part was the PGP signature. It’s probably encoded code that will send someone to nasty sites, or something similar. The sad thing is that this one will probably fool a lot of people who don’t know any better.

A notice to those that don’t know any better… If you get this email, don’t execute the attachment. It will do ugly things to your windows computer, in all likelihood.

Blinky Lights

It should be the goal of every good geek to build something with lots of blinking lights. I’m proud to say that I’ve done this many times over. The latest…

Too bad the system this set of drive arrays is attached to is causing me no end of grief.

Ten Times

I’m really tired of needing to apologize to Comcast. They’re evil. I know this. But I’m really getting annoyed with having what I perceive to be their problems actually be problems on my end.

I’ve been having a lot of problems with random cable modem disconnects due to lack of signal over the past few weeks. I know this because I’m a dork and wrote a script that plugs into my Cacti installation and logs the signal levels displayed in my cable modem’s admin interface. I did this years ago because – you guessed it – Comcast is evil and their stuff cut out on me enough for me to want to be able to monitor it. There has been a steady and constant trend of ever-dropping signal levels that has made itself apparent since the beginning of the year. I’ve had the same general layout in the apartment in terms of devices and number signal paths, so I could only assume that the problem was upstream on Comcast’s side. I was so confident that this was yet another Comcast screw-up that I ordered DSL.

Well, it wasn’t.

I had a service appointment scheduled with Comcast on Friday. The guy did some tests outside where my line plugs in, and he said he noticed a lot of signal “reflection”, and said that my splitters and/or wiring were probably bad. He looked around, and also noticed that I was running my cable through the Monster way-too-much-money power strip, and called that a no-no as well. He replaced three splitters with two nicer ones, and ran custom length cables to each one of my devices. The result?

[signal graph]

As you can see, the signal levels improved dramatically. The downstream power, which is what I receive from Comcast, is up by at least 10dBmV (depending on the time of day). 10dB corresponds to a ten-fold signal power increase. Signal-to-noise ratio is also better than it was, but not by as striking of an amount.

What’s are the lessons here? Use good splitters. Use good cable. Don’t assume Comcast is at fault because they’re evil.

EDIT: My dad has corrected me. The 10dB increase probably only corresponds to around a 3.1 fold increase in power. I think I read the wikipedia chart wrong.

Xen + AoE + drbd = New Redundant Hotness

A few weeks ago, my buddy Justin posed an interesting problem, one that I’ve been pondering myself for some time. He’s somewhat of a Xen zealot like myself, and is doing some Xen setups that are similar in construction to mine, with a central shared storage array and two or more dom0 machines where the child instances will live. The prospect of migrating domUs between dom0s is quite appealing to him, but he, like myself, realized a critical flaw in the setup. If the storage array fails or requires uptime-affecting maintenance of some sort, the whole setup grinds to a halt. That doesn’t really fit the goals he and I are both after.

After a bit of thought, I looked to a project Justin had mentioned a few months back called drbd. It’s designers deem it “network raid 1”, and that’s a pretty accurate description. It’s essentially a system that mirrors data between two different machines either in an active-standby or active-active configuration. One of its primary goals is to provide storage as close to 100% of the time as is possible. Its usefulness would vary highly depending on the application. Having a normal file system on it shared between two machines could be nothing short of a nightmare, since neither server knows what the other is doing until changes are already written to the shared storage. A clustered file system would work well with it though. As I began to learn more about how it works, I realized it could potentially be a great solution for my predicament. Since either of of the two machines could provide storage at any given time, it would have no problem fufilling the near 100% uptime requirement.

What really makes the solution stand out to me isn’t just drbd itself, but the combination of drbd and AoE. AoE is, by design, a connectionless protocol. When the kernel module is loaded, it does a device discovery to see what devices are available for its use, and listens thereafter for new devices. The information it learns is pretty much limited to a MAC address where the storage device is located and the vblade “addresses” within that device that are available. There’s nothing within the protocol that outlaws multiple targets from advertising the same vblade “address”, and it’s up to the AoE initiator in the kernel module to choose where it’s sending data. Because of this, you could have two linux vblade targets running on both “ends” of a drbd setup, and there’d be no conflicts whatsoever. The recommended setup in drbd is to consider a write operation as finished only when data has been written to disk on both “ends” of the drbd setup. Combine that with the fact that AoE will only send commands to one MAC address at a time, and its pretty much guaranteed that both vblade targets will be connected to the same data at all times, even though they’re on different machines. I can think of a scenario or two where data would be out of sync, but it would require that disk write operations be done in parallel, and I’m farily certain that they aren’t.

The fact that the same data is on both machines and that AoE allows for a quick and painless transfer between vblade targets is what makes this such a simple and effective solution for me. There may be a few seconds of lag while the AoE initiator realizes that the machine it was talking to has disappeared, but that will pass as soon as it does another device discovery and sees the other vblade target. This is perfectly acceptable in my usage scenarios thus far.

I took the plunge a week or so ago and started converting my storage at home to use drbds. It’s was pretty simple to convert from my LVM-based setup, since all I had to do was create another single LVM for every partition I wanted to sync between machines. These additional LVM partitions store the metadata that drbd uses to track changes and to keep things in sync. This configuration also allows me to revert back to using “naked” LVM partitions as my vblade storage targets if I decide I don’t like drbd in the future. I used my MythTV recording backend as the second drbd server, since it has a lot of space for extra drives and is on pretty much all the time. I put in a 120GB drive, and let everything fly. Once the initial synchronization was complete, I did a few tests, and everything worked as intended. I could kill vblade targets on either machine, and after a few seconds, the initiators would look at the other machine and use it for storage. Success!

As of this weekend I’ve also converted my setup at work to use the same general configuration. The primary storage consists of a big RAID array, with a secondary machine using a single drive as a backup. I figure that in most cases running in an active-active setup wouldn’t be necessary, so I’m going to stick with active-standby, and only start the vblade targets on the secondary machine when I’m planning on a reboot or other maintenance event. I’ve also considered running in an active-off state (with periodic resyncs), so that there wouldn’t be any performance hit from waiting for the second server to complete its writes. This would probably be a less desirable setup since the data could (and very likely would) be out of date if I were to suffer an unplanned outage such as a hardware failure. Nothing I run currently is terribly needy in terms of disk write performance, so I’m not terribly concerned at this point.

Oh Yeah, This Thing.

It's been a pretty crazy couple weeks, for sure. I've been super busy with multiple projects at work, with each one demanding lots of attention. I tend to have a hard time changing directions once I get going on something, so having so many things to do can be pretty stressful. In addition, a system I take care of at work also had some kind of a nervous breakdown, and required a ton of attention in order to correct. A ton of attention while I was on vacation. Totally weak. It's mostly fixed now though, so it's back to being pulled in multiple directions by other stuff. Sigh.

There hasn't been a lot going on outside of work recently due to a general lack of energy and motivation to do anything. All of the stuff happening at work leaves me with no desire to do anything when I get home. It's pretty depressing, in a way.

One relatively fun thing happened though. For the first time since around 2000, I built myself a brand new computer. I got myself a quad-core AMD Phenom processor clocked at 2.3GHz, 4GB of PC2-8500 RAM, a Biostar TPower N750 motherboard, a power supply, and new case from Newegg. I also got a couple of SLI-capable Nvidia GeForce 7600 GS video cards from a coworker. Combine all the parts together and I've got myself a pretty beefy machine. A hell of a lot more beefy than my other desktops. I've been installing some of my old games on it, and they absolutely scream.

Nice One, Stupid

Even though they're an over-priced pseudo-monopoly with a track record of shitty customer service and only somewhat better service uptime, I owe Comcast an apology. My internet was down for most of the past weekend, and for most of Monday as well. I figured it was due to the storms that rolled through on the night of the outage (Saturday), but after a couple of days with no service, I started getting mad. I was cursing their name and anything related to them. I was particularly unhappy when I found out that a coworker that lives in my apartment complex had no interruptions in service. After I heard that, I started thinking of ways that my setup would be sabotaging the process.

And then it hit me. As part of my process to convert my firewall machine into a Xen instance, I altered the physical networking layout so my cable modem would plug directly into my “Core Switch”, an old Cisco 2924XL. I gave the cable modem service its own VLAN, which would be accessable via my firewall instance running on a Xen machine. What I failed to consider is that managed switches tend to have features that allow for communication with other switches in order to facilitate ease of management and network health. This communication is typically broadcasted to any device that is listening on regular intervals.

These broadcasts are what caused my issue. In a normal residential cable modem service (with Comcast at least), the cable modem latches on to the first network device it hears traffic from, and assumes that it will be the one it deals with when connecting to the internet. By having my cable modem plugged directly into the switch, it was receiving the switch's broadcast messages before my firewall instance had a chance to make itself heard. Because of this, my firewall's attempts to connect to the internet fell on deaf electronic ears.

This was remedied easily enough by disabling spanning tree protocol on the VLAN that my cable modem connects to, and disabling Cisco Discovery Protocol broadcasts on the port it connects to. I don't like disabling spanning tree, because quite frankly, network loops suck. The chance that somehow make a loop in that VLAN is pretty damn low though, so there's not much to worry about.

Let this be a lesson to those with way too much time on their hands, like myself.


I had the camera out the other night, and snapped some pictures from the balcony. They turned out pretty cool I think.

[bird in a tree]

[a tree silhouetting the sun]