Oak Ridge National Lab (ORNL): Oak Ridge Leadership Computing Facility

Background

The Oak Ridge Leadership Computing Facility at Oak Ridge National Lab (ORNL) is home to Summit, a high-performance computing machine. After its launch in 2018, Summit was recognized as the world’s fastest supercomputer. With the capacity to perform 200 quadrillion calculations per second, not only is Summit more powerful than its predecessor, it is also more energy-efficient.

A building-wide retrofit to the 9,400-square-foot data center housing Summit started in 2015. During Summit’s installation, engineers focused on developing an energy-efficient cooling system that could match the supercomputer’s 13 megawatts of energy consumption. Ensuring optimal cooling for a data center is a top priority for operators as it lowers operational expenditures, reduces the strain on equipment cooling mechanisms, and further extends the lifespan of hardware.

Solutions

Moving away from traditional air cooling with chilled water, Summit’s data center features an innovative, warm-water cooling system, reducing the use of chilled water and the reliance on inefficient chillers. Compared to air cooling, liquid cooling has a higher ability to capture and remove heat. Once excess heat has been transferred to a liquid, it can be removed from the data center more efficiently.

Due to space constraints, a single piping system was selected to serve both the cooling and heat exchanger processes. Water runs through the piping at a temperature of 70°F allowing for chiller-less cooling throughout most of the year. Water flowing through the system’s cold plates captures excess heat from each server’s central and graphic processing units. Residual heat is then captured by rear-door heat exchangers (RDHx). Hot air is forced from the server-rack through the RDHx device where excess heat is exchanged from air to water. The wasted heat that has been transferred to water is then moved to a cooling plant before the process starts over.

Some of the other energy efficiency measures implemented include:

LED lighting and timer system
Epoxy-coated pumps to increase pumping efficiency
480V racks to prevent excess electrical transmission losses
Variable frequency drives (VFDs) on motors to align cooling capacity to load
Variable refrigerant flow system that filters and dehumidifies the room air while removing any residual heat released into the room from IT equipment if needed

Looking ahead, facility engineers project that as Summit enters full production its PUE will continue to improve, likely leveling off at an average below 1.10. So far, the upgrades have led to an annual cost avoidance of $430,000.

Oak Ridge National Lab (ORNL): Oak Ridge Leadership Computing Facility

ANNUAL ENERGY USE

ANNUAL ENERGY COST

ANNUAL WATER USE

ANNUAL WATER COST

ANNUAL PLANT ELECTRICITY USE

ANNUAL PLANT ELECTRICITY COST

IN BUILDING ENERGY USE

IN BUILDING ENERGY COST

ACROSS NETWORK USE

ACROSS NETWORK COST

ANNUAL EMISSIONS

USE

COST

Oak Ridge National Lab (ORNL): Oak Ridge Leadership Computing Facility

Related Content