Liquid cooling and heat re-use experiences Gert Svensson Dep. Director PDC, KTH November 11, 2015
Contents PDC Overview Heat re-use at PDC Liquid cooling Case studies
KTH and PDC KTH Royal Institute of Technology is the largest technical university in Sweden. is the largest supercomputer centre in Sweden located at KTH. STOCKHOLM 2015-11-11 3
PDC s Mission Operation of world-class: Supercomputers Data storage & long time data archive Provide world-class: User support Training Education mainly to Swedish academia Take part in world-class research in the area
Funding Part of SNIC (6 centers in a meta center) Most funding from the Swedish Research Council (VR) and KTH Many EU projects Some industrial collaboration (mainly Scania) Note: Academic use is free
How is a supercomputer build today? Cluster Nodes Similar to computer without keyboard and monitor Processors Cores
PDC Main System: Cray XC40 Beskow STOCKHOLM 2015-11-11 7
Cooling at PDC background Listed buildings KTH located between the Royal national city park and the city centre District cooling since 2004 Environmentally friendly (sea water and central heat re-use) Has been fairly expensive but this has changed recently Backup cooling needed 8
Energy consumption at PDC
First Heat Re-Use Project 2009-2014 Cray XE6 600 kw Retrofitted an air cooled system with water cooling Heating of one building (the Chemistry Lab) Heating the incoming air to the building 10
Collecting high enough temperature Our computer room air condition (CRAC) units takes water input of 8 C and produce 18 C Room temperature normally around 20 C CRAC units not sufficiently high temperature The Cray XE60 computer two versions: Water cooled (too low temperature) Air cooled (air output 35 40 C) Decided to purchase the air cooled version and design the cooling ourselves
Cooling the computer Hot air collected by industrial air water heat exchangers Custom made chimneys attach the computer racks and the heat exchangers
Air Air Water 16 C Water ~30 C Air-Water Heat Exchangers Air 21 C Air 21 C 35-44 C 35-44 C Server Racks Air 16 C Air 16 C From: Computer Room Air Conditioners CRAC)
Building to heat Nearby chemistry lab undergoing renovations Mainly heated by air Needs larger amount of air because of chemicals
Transporting water to the chemistry lab Use existing pipes from the district cooling system Change direction of flow when in re-use mode PDC PDC Heat Recycling Loop (winter) Chemistry Building
Water distribution One heat exchanger for re-use One for district cooling 8 C CRAC 18 C PDC Computer Hall 19 C Cray 33.5 External Heat Exchangers C If everything fails use tap water 6 C 16 C Tap Water 19 33 16 District Cooling System C Always external supply only in this section C C Own supply (winter)
IRL
Lessons learned Possible to build something custom-made but lot of work Keep it as simple as possible Re-use existing infrastructure Different perspective in computer industry vs. building industry Requires cold climate but Heat can be made into cold using absorption chillers
Evaluation Winter 2012-2013 Recovered power (kw) Outdoor temperature ( C) kw 500 450 400 350 300 250 200 150 100 50 0-15 -10-5 0 5 10 C 28-Mar-13 21-Mar-13 14-Mar-13 7-Mar-13 28-Feb-13 21-Feb-13 14-Feb-13 7-Feb-13 31-Jan-13 24-Jan-13 17-Jan-13 10-Jan-13 3-Jan-13 27-Dec-12 20-Dec-12 13-Dec-12 6-Dec-12 29-Nov-12 22-Nov-12 15-Nov-12 8-Nov-12 1-Nov-12 25-Oct-12 Saved 540 MWh district cooling Saved 270 MWh district heating Saved 50 keuro per season Saved 50 000 kg CO 2 per season 19
Second heat re-use project New Cray XC40 700 kw Water cooled Central heat re-use facility with heat pump 20
Cray XC40 water cooling (hybrid cooling) CPU board is air cooled but the air is cooled by heat exchangers and fans in the adjacent racks Simple solution but with lower water temperature and some energy loss by the fans (more like cooling doors) 21
Cray XC40 water cooling 22
Central heat re-use at KTH
Experience Supercomputers excellent heaters Almost constant heat Heat re-use really works and pays off Energy to KTH reduced 23,700 MWh/year (25 %) CO 2 reduced 890 000 kg/year (44 %)
Why liquid cooling Liquid cooling is being adopted for a variety of reasons: Silence (No fans) Resilience (More stable, efficient cooling) Efficiency (Fans replaced by pumps) Environment & Cost Higher temperature -> More free cooling Higher temperature -> Possibility for heat re-use Total Cost of Ownership (TCO) Hostile or Complex Environments (Not depending on clean air) STOCKHOLM 2015-11-11 25
Different types of cooling Air Cooling Indirect Liquid Cooling Direct Liquid Cooling Total Liquid Cooling Air is passed through servers and then through a rear door or in row air-water heat exchanger. Liquid is taken direct to some components Fans are still needed. All components are cooled directly by liquid. Air side losses are minimised. Inside the DC, the system has no fans and breathes no air. STOCKHOLM Slide courtesy by Iceotope 2015-11-11 26
ASHRAE Water Cooling American Society of Heating, Refrigerating & Air-Conditioning Engineers
Direct Liquid Cooling Widely Available Cooling with liquid more or less close to the CPU/memory etc. Advantages: Low extra energy used for cooling (liquid pump) Efficient cooling possibility to run CPU faster Depends of how close to CPU Higher liquid temperature than going through air Free-cooling for longer time of the year Possibility for heat re-use 28
PUE is not that important! Heat re-use gives more savings than improving PUE
Problems and some solutions Corrosion and bacteria in hot water Filters, devices to take away air, chemicals, black magic Computers short life time (4 years vs. 10-30 for infrastructure) Lack of standard for liquid cooling Different temperatures and pressures Different requirements on clean water Flexible solutions required!
Experience High temperature is everything Collect heat close to the CPU Best: Direct Liquid Cooling Encapsulate the heat Don t mix high and low temp
Case study: HP Apollo 8000 Closed passive heat pipes on the CPU cards Heat is transferred via heat transfer to a separate liquid loop in the rack 32
Heat Pipe Demo
HP Case study continued 34
HP case study continued 35
HP case study. Modularized plumbing system 36
Another liquid: Mineral oil Green Revolution Cooling Fluid is dielectric Electronics can be submerged in the fluid (not moving parts) One phase Heat is transported by circulation High temp possible (60C) 2015-11-11 STOCKHOLM 37
Yet another liquid: 3M Novec Fluid is dielectric Electronics can be submerged in the fluid One phase Heat is transported by circulation Novec 7300 with boiling point 98 C is used Two phase system Boiling and Condensation Novec 649 with boiling point 49 C is used 3M claims the fluid is environmental safe (but I doubt) The fluid is very expensive
Very simple submerged cooling Interconnect company doing cooling on the side Box of Intel Phi with Extoll interconnect Boiling Novec Water cooled coil
2015-11-11 STOCKHOLM 40
Iceotope
Iceotope: Secondary Coolant flow 45 degrees in Up to 55 degrees out
Rack with highest power RSC 400 kw/rack 1.2 PFLOPS 1024 Intel Xeon Phi 5120D
Liquid cooling activity Lot of products available but not much standardization Water temperatures Water quality Connectors ASHRAE is doing some standards 44
Questions? gert@pdc.kth.se 45