Complete Overkill

This past saturday I spent the whole day at work. Over twelve hours. The day was boring for the most part, tedious, and I even fell asleep once or twice at my desk. Not good you say? The thing is, I wasn’t working. I was migrating data off my personal RAID array so that i could expand it to encompass the second RAID cabinet that I just finished acquiring disks for.

This whole saga goes back to the beginning of the summer. I was in the middle of a pretty large eBay binge, all the while buying tons of compter stuff I didn’t really need. While I certainly didn’t need the stuff, having it around has allowed me to further my knowledge in things like Xen and play with other cool technologies like ATA over Ethernet (AoE). Anyway, during my browsing, I came across a RAID cabinet completely loaded out with 18GB drives pretty much identical to one I had purchased earlier on. The price was pretty cheap, so I nabbed it.

The customary week passed while UPS ground transported the 100 pound-plus package from its origins on the west coast to its new home in Lansing. When it arrvied, I eagerly opened up the box and inventoried its contents, only to find that I wasn’t given all that I was promised. The RAID cabinet itself, the full compliment of twelve 18GB drives, and the mounting rails were all there, but the two external SCSI cables were missing. I promptly got in touch with the seller, and he assured me that it was just an oversight and that he would send the cables right away. Right away turned into a couple of weeks. I finally received the delivery, only to find that he had sent the wrong cables. They would easily fit into the ports on the back of the RAID card in my server, but the other end had the wrong connector type, and wouldn’t connect to the RAID enclosure. I got back in touch with the seller again, and he again promised to right the wrong, and at no charge. Well, those cables never arrived. I was ticked, but I got over it.

Over the next few months I replaced the 18GB disks with 36GB disks so that I could merge the second RAID cabinet into my current RAID array, which is comprised of 36GB disks. Used 36GB drives are relatively inexpensive on eBay so I was able to get the requisite number of disks without too much expense. I also ordered the cables that I was denied in my original purchase.

All the pieces were in place last week, so I decided to try getting everything set up. I got everything put together and connected this past thursday night, and started the process of finalizing things. The RAID card I’m using in the machine (a Compaq/HP Smart Array 5304-128) supports live expansion of an array onto new disks, so I fired up that process from the command line array management utility. I was mildly surprised that I could only extend onto seven of the twelve disks in the second RAID cabinet. I figured that it was just something in the way the expansion algorithm worked, and that I’d be able to expand onto the other disks when the initial expansion finished.

Rearranging 500GB of data on disks is not a quick process, so I had to wait a good twelve hours before I would discover that I was not able to expand the array onto those remaning five disks. I searched around for possible reasons why I was having difficulty and found nothing concrete. I did come across a blurb somewhere that mentioned RAID6 allowing for more disks than RAID5 in an array, which didn’t really seem correct to me, but I decided to try converting the array to RAID6 anyway. Another twelve hours later, I discovered that didn’t work either.

I decided to completely redo the RAID array from scratch, which leads us to last saturday. I was up pretty early, so I made myself a big breakfast and headed into work to do the deed. I got started copying the data off the RAID array, which was no quick process. It takes a while to copy over a hundred gigabytes of data. During this time, I played some games on the laptop, walked around the datacenter, chatted with people, and took a few naps. There’s a surprisingly comfortable position I’ve discovered on my desk where I can just zonk right out…. when I’m not working of course! Anyway, once the data was copied off, I recreated the RAID array and was glad to see that I could use all the disks to their full capacity.

I began the process of copying data back onto the array, and settled in for another nap. I awoke a short time later to find that one of the drives in the array had croaked. It turned out to be one of the disks in the newest group I had purchased, and not really tested all that thoroughly. A disk failure sounds bad, but in RAID6, up to two disks in the array can fail without any data loss. So, in this case, it was just an annoyance. The RAID card detected the failure and immediately started working to rebuild the failed disk’s data on one of the hot spares I have configured. This process slowed down the data transfer pretty substantially, so I was there for a lot longer than I wanted to be. I replaced the failed disk with one of my leftover disks, and the array was happy again.

So, now I have a nice RAID6 array composed of 26 36GB SCSI drives and two hot spares. It weighs in at around 860GB total, and takes up 11U worth of space on the racks at work. I find it pretty funny that there are now single SATA drives that can store more data than that entire array, but they are nowhere near as cool. I have the blinky light factor on my side, as evidenced here…

[RAID setup]

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>