Image via Wikipedia

Word going around is that the State of Utah is looking at possibly consolidating some data centers. In government, over the years, state agencies built data centers and ran them independently. A lot of these were really just machine rooms with not much in the way of power and air conditioning. Some were full-on Class A data centers.

As State CIOs have looked for ways to save money, data center consolidation was a favorite example of how more interagency cooperation could result in tax payer saving. After all, the DNS server for Tax doesn't really care if it's running next to the Web server from Commerce. It's all bits. Oregon, Michigan, Texas, Wisconsin and Georgia are all among states that have taken on consolidation projects--with varying degrees of success.

Over the last ten years there have been a lot of changes in data centers--many driven by changes in the density of the machines that run in them. No longer is space an issue. Rather the issue becomes one of how much power you can get in and how much heat you can get out. Another trend changing data centers is virtualization.

Consolidation can save money if it's done correctly and with a clear understanding of costs, benefits, potential pitfalls, and likely future needs. Costs come from a number of factors:

• Air handling - old data centers may have inefficient air handling units and poor design leading to excessive costs.
• Power conditioning and distribution - older data centers may be inefficient at providing power or may not do a good job of conditioning and providing backup power.
• Facilities management - managing a building costs money. Reusing space for other purposes saves the money needed to buy or rent new space.
• Personnel - older data centers and the applications running in them may not be instrumented for remote monitoring and operation and thus require personnel onsite.
• Security - data centers need physical security to protect the machines inside of them.
• Ping - running multiple lines to multiple data centers may cost more than running a few larger lines to bigger data centers. The need for redundancy multiplies this.

There are also plenty of potential pitfalls. Here are some areas where people run into trouble:

• Undersizing is common - Many data center plans don't make provision for enough space, power, and cooling.
• Relocation is hard - moving servers can mean downed services. More problematic is that services that have grown acretively over the years have lots of interdependencies.
• Baselines are ignored - If the reason for consolidating is to save money, you'd better know how much you're spending before and after so you can understand what happened.
• Too many changes happening at once - consolidations look like a good time to drive other changes, like virtualization, as well. Consolidation is complex enough without adding more moving parts to the mix.
• Experienced hands are unavailable - Not very many people have experience in consolidating data centers and yet it's vital to success.

Here's a few things--by no means exhaustive--that you can do to mitigate some of these problems.

• Consolidate in phases - consolidate servers using virtualization and blade technology before you consolidate data centers so that you have a better idea of size requirement and interdependencies.
• Don't upgrade and move simultaneously - Don't try to upgrade or change systems as part of the consolidation. Freeze systems to be moved in the months beforehand and move the systems as they are.
• Pick low hanging fruit first - concentrate initial efforts on machine rooms and racks in closets. Consider leaving good data centers that can be remotely managed until the end.
• Understand security and privacy requirements of all players - Many agency programs have particular, legally mandate, requirements. Health and education are two that come to mind.
• Don't promise that you'll eliminate servers--at least not at first. That will increase pressure to change systems at the same time you're consolidating.
• Don't underestimate the emotion attached to servers--people like to be able to see their servers. I used to be this way, so I understand it. They're cool. Blinking lights give the impression that important, cool things are happening. Get them over this by moving some things to the cloud or another data center before you take it all away.
• Don't try to get down to one massive data center--multiple, smaller data centers may be cheaper to operate than one large one as long as they can be remotely managed. The model is a small number of smaller, high watt per square foot data centers and one NOC.

One question that seems to come up is what should be outsourced and what should be kept in house. I don't think there's one answer to this question. But there are some ground rules that can help make the decision:

• You should definitely get outside help to plan the consolidation. Experience can make a world of difference.
• Backup data centers and redundancy is a great thing to outsource and often can be placed somewhere that yields additional benefits (like being in a different earthquake zone or where power is cheaper).
• While some system might need to reside on State premises for security or legal reasons, many do not and could be somewhere else--even in the cloud.
• Using consolidation as an excuse to build a new shiny data center can add additional scrutiny. Outsourcing takes away much of the spotlight of a new construction project.
• Make sure that you understand the true costs of running things yourself so that you can make reasoned decisions instead of emotional ones.

Surely, I've forgotten some things that ought to be said. Add your comments below if you have other ideas about how to go about data center consolidation.