The Genome Data Center has received a Gold LEED Certification from the U.S. Green Building Council. This is in addition to the Keystone Award from the St. Louis Association of General Contractors. It is quite an achievement for a power hungry data center to receive a LEED certification, much more a Gold Certification, but the WUSM Design and Construction team along with the architects, engineers, and contractors were able to pull it off.

Recently the final phase of construction at the Genome Data Center was completed. The initial build out had enough power and cooling for about 40 racks of equipment. Now at full capacity, the data center is capable of supplying 4 MW of power (about the amount used by 800 homes on a hot day) and the requisite cooling to the equipment housed within it. This will support over 100 racks worth of high-density computational (blades) and storage equipment and its supporting infrastructure (chilled water plants, air handlers, humidity control, office space, etc.). The electrical system is completely redundant, all the way to the double-ended substation of our electrical utility. That means even if we lose one entire electrical feed, we can still operate on utility power. If we lose both electrical feeds, we have battery and fly-wheel UPS systems to carry us until the two 2 MW diesel generators start (under 10 seconds). The building is about 1480 m² while the actual data center is about 288 m² (as they shrink computing equipment, the required electrical and cooling equipment keeps increasing in size). The data center is arranged in a standard hot aisle/cold aisle layout with cooling delivered from below through floor grates (perf plates did not provide enough airflow) via a 1.2 m raised floor. We currently have about 3,000 cores in our computational cluster and over 3 PB (3,000,000 GB) of storage online. When full of equipment in a few years, the data center will likely house tens of thousands of cores and on the order of 100 PB of storage.

There are more pictures of the Genome Data Center on Flickr.

Well, I took a bit of a jab at St. Louis Magazine for being late to the party on reporting about the first cancer genome sequence, so it is only fair that I also poke a little fun at Outlook, a quarterly publication of Washington University in St. Louis School of Medicine, for waiting until their Spring 2009 issue to report on the AML genome: Dangerous Transformations. I guess they are a quarterly and deserve a little more slack, but surely it could have been in the winter issue. With the recent completion of the second AML tumor and normal genomes, perhaps it is true what they say, “everything old is new again”.

It is worth noting that the picture in the article (linked below) of Rick Wilson, Elaine Mardis, and Tim Ley was taken in our data center. Or as Tim calls it, the holodeck.

You can see more pictures of the data center on flickr. Now that we have moved in, I will post some more pictures as soon as I can.

Here’s a slightly off topic question. My company is looking at doing some engineering for a computer data center. We are talking power to the data center, electrical engineering. The Engineer just came to me and was wondering if there are any guidelines out there that give suggestions from an IT standpoint. He has suggestions from the building engineer. They have supplied a max kwatt requirement. The question is what is a reasonable power requirement? The max kwatt requirement would not necessarily be needed all the time so the need is to try and come to an understanding of what an average requirement might look like or how to go about finding it. Anyone have any ideas or suggestions on this? Are there any documents or standards that suggest power requirements to data centers?

Depending on your equipment, the power numbers could vary wildly. For an entire rack of blades, the power usage could be over 20 kW. High density storage can be 8 kW per rack or more. Tape robots and network switches don’t use much power at all. You need to project (guess) what the equipment population will be when your data center is fully populated. Since the population of a data center normally occurs over the span of years and technology changes radically in that time scale, this is quite challenging. You also need to consider redundancy and the power needed to run the cooling equipment. If you want full redundancy, you need to double everything, realizing that when things are operating normally, each side will only be drawing about 40% of its total capacity (half of 80%, giving you a 20% efficiency cushion). Ultimately, there is so much uncertainty in just estimating what equipment will be put in the data center over its lifetime that trying to estimate how much power each piece of equipment will use on average is highly speculative.

You can find the thread on the mailing list archives (username and password here).

On the plus side, the data center now has its raised floor (currently covered with craft paper to protect the finish during the remaining construction). While this may be good news for us, it is bad news for the electrician who still has some work to do under there.

You can find more pictures on flickr.

Now there are some ways to reduce the size of the supporting equipment. On the electrical side, you can go with fly-wheel UPS systems instead of battery UPS systems. The plus side is that fly-wheel systems are a little smaller and require less maintenance. The down side is that fly-wheel systems only supply energy for 20-30 seconds (so you better have a good generator) and they have higher purchase prices (although lifetime cost may be lower due to the higher maintenance cost on batteries). Unfortunately, the generators, transformers, power distribution units, etc. are just huge.

On the mechanical (cooling) side, there is actually an alternative cooling strategy we investigated that could result in significant space savings. The design that was chosen is basically an amped up large building cooling system, i.e., chilled water. So you basically have large air handling units, chillers, cooling towers, tanks, and lots of pumps, pipes, and valves. This was chosen because this is what the designers, construction managers, and facilities teams were familiar with. An alternative design uses a system with compressor-based computer-room air conditioners (CRAC or Liebert units) coupled with refrigerant-based point-cooling attached to racks (blades generate a lot of heat). The CRAC units have condensers in them and require heat dissipation on the roof. The point-cooling systems require a condensing system outside the data center and all the units require heat dissipation units on the roof of the building. So while you need some stuff outside the data center, a lot of it is on the roof and it is much smaller than the very large chilled water plant (chillers, pumps, cooling towers, air handlers, and tanks). On the other hand, you end up with a lot more systems to maintain (instead of three chillers, you have 20 or so condensers). There are some pluses to having lots of systems. When you have lots of systems, N+1 redundancy costs a lot less than when N is small. You also have lower initial costs because the cooling system is only as large as it needs to be initially, i.e., you don’t have a bunch of extra capacity on day one when only a quarter of your data center is filled. You also don’t have to buy a big, complex, expensive control system, since the CRAC units have controllers built in (and are able to be networked to work together).

If you want, you can see more pictures of the data center construction.