Air cooled servers blow heat generated by all of the components within a server into the data center and rely on computer room air conditioning to remove all of this heat from the data center. Processors (CPUs and GPUs) generate more heat than any other single component in a server. Most of the heat put into a data center by servers typically comes from the processors.  Using liquid cooling to bypass the CRAC system and remove processor heat directly from the data center enables substantial power savings. This is exactly what Rack CDU does.

Data centers with Rack CDU use two cooling paths to remove heat from the data center. As shown in the illustration to the right, a traditional air path with CRAC units removes heat generated by disk drives, power supplies and other low heat flux components. On the right side of the illustration, the Rack CDU system uses a direct liquid cooling path to remove 100% of the processor heat from the data center. Two CPUs are liquid cooled in this illustration; however any combination of CPUs and GPUs can be cooled. Direct liquid cooling of memory is also available for servers with large amounts of memory.

The liquid path used by Rack CDU systems is made up of two separate liquid loops, a low pressure server loop (lower loop in illustration) and a facilities loop (upper loop in illustration). No liquid passes between these loops. Facilities water never enters the servers, instead heat moves between the loops via a liquid-to-liquid heat exchanger in the Rack CDU. Separating the loops at the rack level enables the Asetek server loop to operate at such low pressures that a pressure spike leading to a failure within a server simply cannot occur. Additionally, it tightly limits the amount of liquid that is available for spillage within the rack should the loop be damaged.

The server loop runs through each server and connects in parallel to a hot/cold liquid distribution manifold within each rack. The hot/cold distribution manifolds are connected via a liquid-to-liquid heat exchanger and a reservoir.  Low power, low pressure pumps are integrated with each CPU cold plate within the server. These are the same pumps that are in service today cooling nearly a million CPUs and GPUs in tower PCs and workstations. The pumps draw cold liquid from the cold manifold and move the liquid through the cold plates and into the hot manifold. Each pump is capable of providing sufficient flow for two cold plates with the second pump stopped, providing N-1 redundancy. In addition, the pump motors are hot swappable.

Liquid from the hot manifold is directed into the liquid-to-liquid heat exchanger by a circulation pump. The liquid-to-liquid heat exchanger transfers processor heat from the server cooling liquid into facilities water, and server cooling liquid returns to the cold manifold. The reservoir also serves as a bypass flow valve, preventing the circulation pump from building up excess pressure.  Each Rack CDU is equipped with a monitoring system that provides leak detection and reports liquid temperature, flow, pressure and level.

The facilities water loop delivers “cool” water to the liquid-to-liquid heat exchanger in each Rack CDU unit. The requirement for Rack CDU is to keep processor below their specified T-case max, which for most server processors is greater than 60°C. A T-case max of 60°C means that the “cool” water delivered by the facilities water loop only needs to be cooled to 40°C or above. Free ambient temperature air is sufficient to cool facilities water to this temperature in most data center locations. In many temperate and cooler climates the facilities water loop can put processor waste heat to use warming buildings.

All components of the server liquid loop are 100% helium integrity test by Asetek and factory filled with cooling liquid. Data center operators (and server OEMs) never need to touch the liquid in or refill the server loop. The portion of the server loop that goes into the server is installed by the server builder. It attaches to the same mounting points as a processor air cooler and the tubes exit the server at a PCIe blanking panel or other convenient location. The manifold mounts in a common rack extension. The servers connect to the manifold with quick connects at the end of tubes which may be routed through normal cable management arms.  Data centers provide facilities water connections for the Rack CDU liquid-to-liquid heat exchanger and installation is complete. Rack CDU retrofits into existing data centers and even to existing servers.

Data Center Density

Simple Example:

• If 1U server draws 500W and its CPUs use 60% of the wattage under load.
• Data center is provisioned with CRAC supporting 8kW / rack.
• Density is limited to 16 servers per rack (8kW / 500W).

With Rack CDU Liquid Cooling:

• 60% of the thermal load (300W) is removed by liquid cooling.
• Each server puts a 200W load on the CRAC.
• Density can expand to 40 servers per rack, a 150% increase.

Data Center Power Savings

Simple Example:

• If 1U server draws 500W and its CPUs use 60% of the wattage under load. A fully populated rack draws 21kW.
• CRAC power draw to support 21kW is 7kW per rack. (Cooling PUE = 1.34).

• 60% of the thermal load is removed by liquid cooling.
• CRAC power draw is 2.8kW per rack.
• Liquid cooling uses 1.4kW per rack.
• $2,433 in net power savings per rack per year (@10 Cents/kWh).
  
  

• 60% less capacity needed in CRAC.
• No server heat sinks, fewer fans.
• Liquid cooling is economizer, liquid plumbing, CDU and cold plates.