Image via Wikipedia

I've been heads down for the last few weeks getting a project out the door for a new customer. As I mentioned, this involves creating a virtual appliance. I decided, due to the circumstances of this deployment that the best option was the build an appliance factory that is capable of churning out new virtual machines at will. I'm going to describe how I did that in this post.

There are bascially three steps to creating a new image that runs the Kynetx Network Service (KNS):

1. Create a new virtual machine
2. Install packages and Perl libraries, create users, and otherwise configure the machine to run KNS
3. Deploy the KNS code and test it

I was exporing Kickstart files for automatically installing Fedora and CentOS when someone pointed me at Cobbler. Cobbler is a Linux installation server that is simply amazing. It includes templated kickstart files, DHCP and DNS servers, the ability to manage multiple distros and repositories, and a database for keeping it all straight.

You start by importing distros and images, then define profiles that combine those with kickstart files, and finally create system definitions for each machine refering to profiles. I pnly needed one distro, one repo, and one kickstart, so I ended up with multiple systems hanging off of one profile. Once that's done, a command called koan (kickstart over a network) is used on the Dom0 machine to create virtual machines as defined by the system definitions cobbler.

I carefully edited the kickstart file to create just the machine I wanted with the right packages installed. At this point, I was building new VMs and taking them down 20-30 times a day as I tested this. That's the beauty of automation--tacking up a machine is just dirt simple.

I was lucky that I'd already invested considerable effort in Puppet recipes for building the environment that KNS need to run, so the second step was almost done. In fact, with just a few edits, I had Puppet building the new VMs up.

The third step was also one that I'd spent some time on. I have a custom deploy script (in Perl) that deploys KNS code based on server role and takes care of all the little details like setting up the configuration files for the various servers.

Every system is slightly different, but I think there's a definite distinction between machine setup, system configuration, and code deployment. The first creates a fairly standard environment, the second configures it to a specific purpose, and the third manages the code.

Some thoughts on all of this:

• Some have asked "Why not put the code in Puppet (i.e. why use a deployment system)?" My answer is that code deployment is a dynamic process that I want more control of than puppet's automatic configuration provides. You could probably press Puppet into this, but it didn't seem to fit for me.
• I had to create a simple YAML-based configuration file for KNS to pull everything together. YAML was the right answer for this. I chose to put that configuration file in Puppet, but I think I'll pull it into the deployment process in the future.
• One missing piece is a database that everything can read system configurations from. Cobbler provides a light-weight one that may serve our purposes for a while, but something like iClassify is more flexible. Right now there's system information in Cobbler, Puppet, and the deploy script. There's a way to put additional attributes in Cobbler that we could use in other places.
• All of this--Cobbler, Puppet, and the deploy script--were installed and running on a virtual machine that we call the factory. That one image, once installed in Xen is capable of creating as many copies of each type of machine we run as needed.
• This can all be done on physical boxes too, of course, but I prefer the flexibility of virtual machines--even when only one will be running on the physical hardware. They can be moved, replicated, and managed with a lot more ease that physical hardware. Plus I have the ability to fire up new ones for QA or whatever without buying and installing new physical hardware. When a 8 core, 32 Gb box costs $4K, you can amortize that investment a lot with virtual machines.

Startups need to be lean. Achieving that goal in a compute-intensive business requires automation. Fortunately with tools like Cobbler and Puppet, automating the build-side of your infrastructure is not only possible, but fairly easy. We manage several dozen machines with only a few hours a week of effort. What's more, adding a new box for load or experimenting is as easy as typing a few commands and waiting 20-30 minutes.