Liquid cooling and heat re-use experiences. Gert Svensson Dep. Director PDC, KTH November 11, 2015

Transcription

1 Liquid cooling and heat re-use experiences Gert Svensson Dep. Director PDC, KTH November 11, 2015

2 Contents PDC Overview Heat re-use at PDC Liquid cooling Case studies

3 KTH and PDC KTH Royal Institute of Technology is the largest technical university in Sweden. is the largest supercomputer centre in Sweden located at KTH. STOCKHOLM

4 PDC s Mission Operation of world-class: Supercomputers Data storage & long time data archive Provide world-class: User support Training Education mainly to Swedish academia Take part in world-class research in the area

5 Funding Part of SNIC (6 centers in a meta center) Most funding from the Swedish Research Council (VR) and KTH Many EU projects Some industrial collaboration (mainly Scania) Note: Academic use is free

6 How is a supercomputer build today? Cluster Nodes Similar to computer without keyboard and monitor Processors Cores

7 PDC Main System: Cray XC40 Beskow STOCKHOLM

8 Cooling at PDC background Listed buildings KTH located between the Royal national city park and the city centre District cooling since 2004 Environmentally friendly (sea water and central heat re-use) Has been fairly expensive but this has changed recently Backup cooling needed 8

9 Energy consumption at PDC

10 First Heat Re-Use Project Cray XE6 600 kw Retrofitted an air cooled system with water cooling Heating of one building (the Chemistry Lab) Heating the incoming air to the building 10

11 Collecting high enough temperature Our computer room air condition (CRAC) units takes water input of 8 C and produce 18 C Room temperature normally around 20 C CRAC units not sufficiently high temperature The Cray XE60 computer two versions: Water cooled (too low temperature) Air cooled (air output C) Decided to purchase the air cooled version and design the cooling ourselves

12 Cooling the computer Hot air collected by industrial air water heat exchangers Custom made chimneys attach the computer racks and the heat exchangers

13 Air Air Water 16 C Water ~30 C Air-Water Heat Exchangers Air 21 C Air 21 C C C Server Racks Air 16 C Air 16 C From: Computer Room Air Conditioners CRAC)

14 Building to heat Nearby chemistry lab undergoing renovations Mainly heated by air Needs larger amount of air because of chemicals

15 Transporting water to the chemistry lab Use existing pipes from the district cooling system Change direction of flow when in re-use mode PDC PDC Heat Recycling Loop (winter) Chemistry Building

16 Water distribution One heat exchanger for re-use One for district cooling 8 C CRAC 18 C PDC Computer Hall 19 C Cray 33.5 External Heat Exchangers C If everything fails use tap water 6 C 16 C Tap Water District Cooling System C Always external supply only in this section C C Own supply (winter)

17 IRL

18 Lessons learned Possible to build something custom-made but lot of work Keep it as simple as possible Re-use existing infrastructure Different perspective in computer industry vs. building industry Requires cold climate but Heat can be made into cold using absorption chillers

19 Evaluation Winter Recovered power (kw) Outdoor temperature ( C) kw C 28-Mar Mar Mar-13 7-Mar Feb Feb Feb-13 7-Feb Jan Jan Jan Jan-13 3-Jan Dec Dec Dec-12 6-Dec Nov Nov Nov-12 8-Nov-12 1-Nov Oct-12 Saved 540 MWh district cooling Saved 270 MWh district heating Saved 50 keuro per season Saved kg CO 2 per season 19

20 Second heat re-use project New Cray XC kw Water cooled Central heat re-use facility with heat pump 20

21 Cray XC40 water cooling (hybrid cooling) CPU board is air cooled but the air is cooled by heat exchangers and fans in the adjacent racks Simple solution but with lower water temperature and some energy loss by the fans (more like cooling doors) 21

22 Cray XC40 water cooling 22

23 Central heat re-use at KTH

24 Experience Supercomputers excellent heaters Almost constant heat Heat re-use really works and pays off Energy to KTH reduced 23,700 MWh/year (25 %) CO 2 reduced kg/year (44 %)

25 Why liquid cooling Liquid cooling is being adopted for a variety of reasons: Silence (No fans) Resilience (More stable, efficient cooling) Efficiency (Fans replaced by pumps) Environment & Cost Higher temperature -> More free cooling Higher temperature -> Possibility for heat re-use Total Cost of Ownership (TCO) Hostile or Complex Environments (Not depending on clean air) STOCKHOLM

26 Different types of cooling Air Cooling Indirect Liquid Cooling Direct Liquid Cooling Total Liquid Cooling Air is passed through servers and then through a rear door or in row air-water heat exchanger. Liquid is taken direct to some components Fans are still needed. All components are cooled directly by liquid. Air side losses are minimised. Inside the DC, the system has no fans and breathes no air. STOCKHOLM Slide courtesy by Iceotope

27 ASHRAE Water Cooling American Society of Heating, Refrigerating & Air-Conditioning Engineers

28 Direct Liquid Cooling Widely Available Cooling with liquid more or less close to the CPU/memory etc. Advantages: Low extra energy used for cooling (liquid pump) Efficient cooling possibility to run CPU faster Depends of how close to CPU Higher liquid temperature than going through air Free-cooling for longer time of the year Possibility for heat re-use 28

29 PUE is not that important! Heat re-use gives more savings than improving PUE

30 Problems and some solutions Corrosion and bacteria in hot water Filters, devices to take away air, chemicals, black magic Computers short life time (4 years vs for infrastructure) Lack of standard for liquid cooling Different temperatures and pressures Different requirements on clean water Flexible solutions required!

31 Experience High temperature is everything Collect heat close to the CPU Best: Direct Liquid Cooling Encapsulate the heat Don t mix high and low temp

32 Case study: HP Apollo 8000 Closed passive heat pipes on the CPU cards Heat is transferred via heat transfer to a separate liquid loop in the rack 32

33 Heat Pipe Demo

34 HP Case study continued 34

35 HP case study continued 35

36 HP case study. Modularized plumbing system 36

37 Another liquid: Mineral oil Green Revolution Cooling Fluid is dielectric Electronics can be submerged in the fluid (not moving parts) One phase Heat is transported by circulation High temp possible (60C) STOCKHOLM 37

38 Yet another liquid: 3M Novec Fluid is dielectric Electronics can be submerged in the fluid One phase Heat is transported by circulation Novec 7300 with boiling point 98 C is used Two phase system Boiling and Condensation Novec 649 with boiling point 49 C is used 3M claims the fluid is environmental safe (but I doubt) The fluid is very expensive

39 Very simple submerged cooling Interconnect company doing cooling on the side Box of Intel Phi with Extoll interconnect Boiling Novec Water cooled coil

40 STOCKHOLM 40

41 Iceotope

42 Iceotope: Secondary Coolant flow 45 degrees in Up to 55 degrees out

43 Rack with highest power RSC 400 kw/rack 1.2 PFLOPS 1024 Intel Xeon Phi 5120D

44 Liquid cooling activity Lot of products available but not much standardization Water temperatures Water quality Connectors ASHRAE is doing some standards 44

45 Questions? 45