I'm working on setting up a new server to host windley.com (including this blog) and my other websites. I've been hosting with Verio. A good fiend of mine started the company that eventually became the Verio hosting division about the same time I was starting iMALL. We traded lots of services and had some partnerships. One of the most personally gratifying was a comped virtual server he gave me in 1996. I've used it happily for seven years. But all good things come to an end and last year they started charging me. The per megabyte charges are getting to me. Everytime I turn around I'm buying another 100Mb. A content management system like Radio that publishes static pages for everything eats up a surprising amount of disk space. Add the fact that I'd like to put more pictures in my album, etc. and I eventually decided I ought to just configure my own server.

Since the server will be running far from my watchful eye and largely unattended, I didn't want to rely on a backup system that required changeable media. So, I decided to buy two 80Gb drives and configure them in RAID-1. RAID-1 gives you mirroring and only requires two drives. RAID-5, for example, requires at least three. Now, I haven't configured a RAID system for a while (five years?) and I was surprised at how easy Linux makes it. You choose the options during the disk partitioning and it just happens. What's not as easy is deciding how, exactly, to partition the disks.

I found a handy tutorial on RAID-1 at LinuxJournal. The tutorial is in two parts. Part 1 "describes RAID, in which cases RAID-1 is useful, the RAID-1 installation requirements and how to install RAID-1 when you have an existing ext2 filesystem." Part 2 shows "how to make a RAID-1 swap device and how to boot from a RAID-1 device, using RAID-1 to facilitate disk backups."

One of the things I learned from the article that I hadn't realized before was that the swap partition can and indeed should be a RAID device since you won't keep right on ticking if you lose your swap on a disk failure. You also want each drive to be on a separate controller and that's easy enough to do. In the end, here's what I came up with:

I made all of these RAID-1 partitions. I created the two software RAID partitions for /boot first so that they'd be in the first 1024 cylinders. I like a separate /var partition so that if log files go crazy due to a DOS attack or something, it doesn't fill up the rest of the disk. /tmp is a link to /var/tmp for the same reason. I like the ext3 journaled file system.

Now for the testing. With this set-up, I removed power from one of the drives and the system gave a few hiccups while it timed out trying to contact that device and then just kept right on going. I rebooted the machine and it came up just fine on one drive. I know I shouldn't do this with the power on, but I couldn't resist. I had to see it work. If you're not prepared to just trash a drive and buy a new one, don't be a hot-swap cowboy.

When you plug it in, you have to use raidhotadd to bring the drive back online. You use it like so:

[root@lynx root]# raidhotadd /dev/md0 /dev/hdc5
[root@lynx root]# raidhotadd /dev/md3 /dev/hdc3
[root@lynx root]# raidhotadd /dev/md1 /dev/hdc2
[root@lynx root]# raidhotadd /dev/md2 /dev/hdc1

You can get the mapping from /etc/raidtab. Recovery takes a while (a couple of hours on an 80Gb drive). This is very disk intensive, so if one drive crashes and the other one isn't sounding healthy, you'll need to cross you fingers and hope you get a recovered disk before the other one blows. Unplugging the other gave the same results. I'm now comfortable that a drive failure won't take down the machine and that I can boot it on a single drive.

One note: if you want the machine to reboot automatically, all of your mountable partitions (everything in /etc/fstab) need to be RAID or else the reboot will die when it can't find a non-RAID partition on the dead disk. You can still boot, but you have to go into maintenance mode and manually remove it from /etc/fstab.

Lastly, if you're going to have a system that masks errors, you need to instrument the system so that you get notified when the errors occur. You could operate for a long time on one drive if you don't. There are some scripts that can be put in the crontab at http://www.1U-Raid5.net/Monitoring/. I like this one.

Its still amazing to me that I can get a RAIDed, journaled file system for free as part of Linux. When we built Emerald Lake we spent lots of money for these same features. Now, they power my blog.

If you have comments about my set-up or decisions, I'd love to hear them. There are plenty of people who know more about this than me. If you think I've made a horrible error, or even a minor one, leave a comment.