Power Savings Running LINPACK Liquid Cooled

Transcription

1 Power Savings Running LINPACK Liquid Cooled Steve Harrington, Ph.D. Lecturer In Aerospace Engineering, UCSD October 24, 2012

2 Introduction Problem: Modern HPC Systems Need Power & Density Solution: Liquid Cooling, direct to the CPU Cooler Processors Result in Lower Leakage Current Less Airflow requires less fan power No Chillers Required Whole System optimization, CPU,Fans,HVAC > lowest kwhr/solution

3 Air Cooled System Heat Flow Path and Cost. Total power to A/C:135 Watts System includes 3 air Heat Exchangers Cost of Cooling: $70/CPU yr 80 watts for A/C and $.10/kwhr

4 Water Cooled System Heat Flow Path and Cost Cost of Cooling: $5/CPU*yr + higher power CPU, + less leakage current + all the GPUs you can provide power for.

5 Liquid Cooling System Details Liquid cooling through the base of finned heat sink Air Back up Cooling when Liquid Cooling is off Indirect Cooling of other server components. Reduce fan speed by 50%, fan power by 75% Negative Pressure, Failure Tolerant Leakproof Simple Modifications

6 Pilot System Functional Diagram Includes Embedded Controller Pump and Heat Exchanger Outdoor Cooling Tower/Chilled Water To additional Servers Heat Sink CPU Flow sensor Fan controller Pump/Reservoir 4U Existing or off the shelf Heat Exchanger 10 kw Cool Flo Module Patent others pending

7 What About Leaks?

8 UCSD Pilot Long Term Multiple Servers 3 Servers Dual E5440 Uses facility chilled water 75% less fan power 60% Reduced HVAC Load 6% less CPU power Running for 4 months Reduced Vibration > better reliability Failure tolerant.

9 UCSD Pilot System Typical data from cold server room 65 F (18 C) Live data at:

10 Thermal Performance, Modified Heat Sink, 6x better than Air Cooled. Thermal Resistance Heat Sink to Water, Dynatron K watts Thermal Resistance C/watt Flow rate ml/min

11 UCSD Pilot Servers Converted to Liquid Cooling Heat Sinks Removed and Replaced By Liquid Cooled Version Tubing Routed. Connector Added

12 Water Cooled Finned Heat sink Cools down Nearby Components Visible Image IR Air Cooled IR water Cooled The Voltage Regulator downwind of the heat sink is cooled by the air coming off the heatsink, even with reduced fan speed.

13 Single Server Tests: Dual E Liquid Cooled. Air cooled at 27C using two different heat sinks for throttled and non throttled operation Water cooled with inlet temperatures from C Tested using LINPACK with warmup, 1,2, 8 and 16 threads Data recorded: Time to complete run, core temperature & speed, Performance in GFLOPS 12V bus power, water flow rate, temperatures of heat sinks, water in, water out

14 Power Savings Due to Reduced Leakage Current

15 Liquid Cooled Heat Removal Percentage of Power Extracted vs Water Temperature, 27 C air Liquid Cooled CPU Heat Sinks 80% Percentage of Server Power Removed by Liquid 70% 60% 50% 40% 30% 20% 10% 0% y = x Water Temperature (C)

16 Air Cooled System Performance 26C Air Cooled Dual E Linpack test Temperature, Speed, and Power Power (w) and Temperatuer(C) avg core Temp Server Power (Resampled) CPU0 (R l d) Speed (MHz) Time (sec)

17 Liquid Cooled System Performance 15C Water Cooled Dual E Linpack test Temperature, Speed, and Power Power (w) and Temperatuer(C) avg core Temp Server Power CPU0 Heat Sink d Speed (MHz) Time (sec)

18 Liquid Cooled LINPACK Speed and Power Inlet water temperature Average Core temp power 12 V DC bus Power AC Total Server Power Savings Time to complete run (sec) Speed increase vs cold water Energy to solve (kwhr) Percentage of DC power extracted with liquid cooling Power Savings due to reduced HVAC Total Power Savings, Server +HVAC % % % 15.8% 28.0% % % % 12.6% 23.4% % % % 7.5% 13.4% % % % 4.4% 7.6% air 2u % 0.77 air 1u (throttlin % U Heat sink 2 U Heat sink

19 Functional Diagram: 1000 Server System Single Cool Flo System in a Standard Rack Cools up to 300 kw

20 Conclusion Reducing Processor Temp from 70 C to 30 C reduces CPU power by 12%, system power by 7% Liquid Cooled Processors Reduces fan power by 75%. No chillers are required anywhere in the world for liquid cooled processors. HPC figures of Merit: GFLOPs and kwhr/solution

21 Thanks Cirrascale Loaned us the dual E server. UCSD is hosting the pilot at CALIT2 SDSC will host our next demo system

22 Backup Slides Prototype System at ChillDyne 4 U Pilot System Cooling Tower Servers, Tubing and Manifold

23 Backup Slides Cool Flo Cooling Layout: Intel Development System In this case the Cool Flo system cools the CPU, Ram and Chipset For Pilot System, Power and Temperature Monitoring is Installed

24 Backup Slides Cool Flo Distribution of Power Savings 120% Power (Normalized) 100% 80% 60% 40% 20% UPS HVAC/fans Lighting/Other Server 0% Air Cooled Liquid Cooled Based on Test Data for Modern Dual E Server Assuming Typical Power Distribution LBNL 2007

25 Backup Slides Cool Flo Removing all the Heat with 2 Heat Sinks.

26 Backup Slides Cool Flo Power Density (W/cm 2 ) It s not Rocket Science. AMD Intel Power PC Pentium 4 (2005) Year Core 2 Duo (2006) Atom (2008) Rocket Nozzle (7000 W/cm 2 ) Nuclear Reactor Hot Plate Flometrics BioFuel Rocket (2009) PopSci Best of What s New

27 Backup Slides Cool Flo Air/Water Cooled Heat Sink Test Data Thermal Resistance C/watt Thermal Resistance Heat Sink to Water Dynatron G watts Flow Rate ml/min

28 Backup Slides No Drip Hot Swap Connector Operation Normal operation The button is depressed half way to block the incoming coolant. Air from the vent replaces the fluid. As long as there is fluid in the valve, the button will not depress all the way because of the incompressible liquid trap. Only once all the liquid in the heat exchanger is replaced by air is it possible to press the button all the way down, unlocking the connector from the aired out server and sealing the coolant behind the valve.

29 Backup Slides Cool Flo Solution Rack based Direct Liquid Cooling High Density and High Power GPUs and CPUs 25 35% Energy Savings Less Leakage Current Less Fan Power Less HVAC Power Server Agnostic Warranted Solution Customer ROI <1 Year Simple Modifications