Operational aspects of the VELO cooling system of LHCb Eddy Jans (Nikhef) on behalf of the LHCb VELO group Introduction Main components and operation principle of the system Issues: how to prevent and to tackle them Keep the detectors cold 24/7 Summary & outlook Eddy Jans 0
Introduction VELO-module is double-sided (300 mm, oxygenated, n + -on-n) and operated in vacuum, Strip closest to the beams is at 8.2 mm, Per double-sided module: 2x16 frontend chips that together dissipate ~20 W, 4 NTCs give temperature readings Two movable detector halves with 21 VELO + 2 PileUp modules each (400 W/side) NTCs Eddy Jans 1
One detector half CO 2 connections module base is kept at +20 o C 0.3 mm thick RF-box Vacuum tank cooling block frontend chips Detectors are operated in a secondary vacuum absolutely no leaks allowed. Orbital welding or vacuum brazing of the pipes. All tested at 170 bars. Eddy Jans 2
Some cooling considerations in 1999 it was proposed to use CO 2 as refrigerant for the vertex detector of LHCb [LHCb note 99-046], CO 2 is radiation hard, CO 2 has a high latent heat value can use small diameter capillaries small amount of dead material in the acceptance, stainless steel capillaries: F inner =1 mm, wall thickness 0.25 mm system uses bi-phase CO 2 via the accumulator controlled method. cooling blocks Eddy Jans 3
pressure [MPa] pressure [bar] liquid phase bi-phase area Pressure x Enthalpy diagram of CO 2 bi-phase area -30 o C critical point isothermal cooling gas phase At -30 o C: 300 J/g mass flow: 10 g/s per module: 0.43 g/s full evaporation 130 W liquid CO 2 -speed: 28 cm/s all-gas speed: 240 cm/s vapour quality 0 1 enthalpy [kj/kg] Eddy Jans 4
Heat in The accumulator controlled cooling cycle Heat out Heat out R507a chillers Condenser Pressure Heat in accumulator P 7 1 5 6 4 2 3 55 m transfer line pump restriction evaporator only passive components in the radiation zone f=80 mm insulation sub-cooled liquid in bi-phase return liquid vapor 1 2: pump increases the pressure of the sub-cooled liquid 2 3: heat exchange in the transfer line brings evaporator pre-expansion per definition right above saturation point, since (E 2 -E 3 ) = -(E 5 -E 6 ) 3 4: pressure drop in restriction and expansion in capillary brings CO 2 in cooling blocks in bi-phase state, 4 5: isothermal cooling via evaporation 5 6: warming up of incoming sub-cooled liquid 6 1: condensation and cooling of the CO 2 P 7 2 3 1 4 6 5 bi-phase Enthalpy Eddy Jans 5
TL transfer line TR transfer line TL CO2 bridge TL CO2 system Main components of the system VELO Velo vacuum tank with silicon modules, module base and CO 2 evaporator. Cooling plant at UXA03 PLC and electronics rack Air-cooled SB chiller Water-cooled SA chiller TLR CO2 system TR CO2 bridge TR CO2 system evaporative CO 2 cooling system independent system for either side PLC-controlled Liquid Pump 2.5 kw water-cooled chiller at 40 o C Electronic 3-way valve 1 kw air-cooled backup chiller at -25 o C 55 Electronic m CO 2-way valve 2 transfer lines 10 Manual heat restriction exchangers valve 8 actuators No return valve 9 heaters Electric heater 31 pressure sensors Damper with heater 192 temperature sensors 350 Heat parameters exchanger monitored in PVSS only passive components at VELO 2*400 W heat load of detectors 2*12 kg CO 2 Performance @ detector main chiller: -28 o C operational CO 2 temp. LV on: sensors @ -7 o C backup chiller: LV off: @ -8 o C stability < 0.1 o C Eddy Jans 6
Design considerations and operational experience redundancy of crucial components insulation clogging filters superheated CO 2 dependence on electrical power dependence on chilled water safety measures to prevent overheated detectors keeping the system 24/7 cold Eddy Jans 7
Redundancy in the design To minimise down time the system has a few redundant crucial components: 3 CO 2 pumps, where 2 are needed, 2 chillers, water-cooled and an air-cooled as backup, for controls crucial temperature and pressure sensors are two-fold implemented, possibility to interconnect the two sides by hand, PLC is on a 1500 VA/1000 W UPS, PLC, backup chiller and CO 2 pumps are connected to a diesel generator. Eddy Jans 8
Insulation Liquid pumped system cold transfer lines good thermal insulation required. This seems trivial, but turned out not to be so in practice. Originally CERN safety regulations forced us to use Armaflex NH. Glued surfaces started to delaminate after 2 years. Renewed insulation of the transfer lines and most of the cooling plant during Winter shutdown 10-11. Now foamglass covered by an Aluminium protection shield and Armaflex AF, respectively. Eddy Jans 9
Filters Throughout the system eleven 15 mm filters are installed. (5 (CO 2 -plant), 2(@VELO), 2(manifold), 1(main chiller), 1(backup chiller)). In one detector half we have experienced a few times clogging filters. pressure [bar] 17 16 15 2 months Replacement procedure is tricky and risk of additional dirt in the system due to difficult accessible filter houses. piece of Armaflex thermal insulation once completely blocked a restriction valve Eddy Jans 10
Post mortem analysis of the filters cps Fe 75 mm 25 20 15 10 O Cr Fe Scanning Electron Microscope image 5 Cr Fe Si Cr Ni 0 0 5 10 Energy (kev) Energy Dispersive Spectrometer analysis of an orange particle found inside the filter Many >15 µm orange objects have been observed inside the filter. They mainly contain Fe and O. Before, particles containing Cl had been observed. Possibly due to connections soldered with flux for a testbeam experiment risk in terms of corrosion. Work extremely clean from construction to installation. Eddy Jans 11
Superheated CO 2 after startup After startup we occasionally observe in a varying number (a few all) of cooling blocks the phenomenon of superheated CO 2. Issue: cooling performance is very bad because liquid cooling has much less cooling power than evaporative cooling. Eddy Jans 12
Superheated CO 2 after warmup and cooldown T silicon [ o C] -14-22 LV off warmup of the cooling plant cooldown DT=3 o C -30 30 minutes time Eddy Jans 13
Pressure T silicon [ o C] -14 Superheated CO 2 after warmup and cooldown LV off liquid 2 3 bi-phase vapor -22 start adding heat 1 4 6 5 Enthalpy DT=3 o C -30 30 minutes time Remedy: add heat by means of a dedicated heater to bring the incoming CO 2 in the liquid+gas state. Eddy Jans 14
Some more superheated CO 2 Not all cooling blocks behave the same way: - not all show superheating - when adding heat they don t start boiling at the same moment silicon temperatures of 4 modules DT 4 o C start adding heat Eddy Jans 15
Power cuts PLC and backup chiller are connected to the power of a diesel generator of LHCb and the PLC also to its own UPS (1500 VA/1000 W) When the power gets cut the switch-over from main to backup chiller is handled automatically by the PLC. -10 sensor temperature After switching back to the main chiller the system is stable after ~20 minutes. After switching on the LV the sensors are at their operational temperatures after 10 minutes. half an hour recovery time. -20-30 10 minutes LV on Eddy Jans 16
Failure of chilled water supply Chilled water supply, that cools the main chiller, sometimes gets interrupted. If so, the PLC switches on the air-cooled backup chiller. However, this causes the LV to be switched off also. Eddy Jans 17
Conditions Vacuum-PLC Cooling-PLC Module temperatures Temp-boards Beam Conditions Monitor 1. HW-based: interlock system Safety Operation in vacuum requires immediate reacting safety systems. Three levels. Actions LV Off HV Off Cooling Off Retract VELO 2. SW-based: warning and interlock system Emergency button Combined information of 4 NTCs per module are input to the FPGA, which can interlock the LV. 132 cooling parameters monitored 3 levels each: warning, error and fatal Eddy Jans 18
3. Human-based Emergency button in the LHCb control room to power off the VELO. Eddy Jans 19
Keep the detectors cold 24/7 At the tip the received fluence is 2x10 14 n eq /cm 2 and type inversion has taken place, so the sensors should always be kept cold, (below -8 o C), in order to prevent the V depletion to increase due to reverse annealing. V depl N C + DN eff (=effective space charge density) Short term: Beneficial annealing (N A ) Long term: Reverse annealing (N Y ) - time constant depends on temperature: ~ 500 years (-10 C) ~ 500 days ( 20 C) ~ 21 hours ( 60 C) The beneficial annealing budget amounts to a handful of weeks at room temperature. We try to save it till we really need it. But the conditions of the LongShutdown1 period at LHCb make it hard to do so. D N eff [10 11 cm -3 ] 10 8 6 4 2 0 N A g C F eq [M.Moll, PhD thesis 1999, Uni Hamburg] 1 10 100 1000 10000 annealing time at 60 o C [min] N C N Y N C0 Eddy Jans 20
The challenge is to keep the system operational 24/7. Under normal conditions the PLC deals with common problems. Goal: minimize the warm time due to scheduled maintenance of crucial components, repair of malfunctioning components, unexpected problems during LS1 and shutdown periods. Eddy Jans 21
Regular maintenance Yearly maintenance of the R507a chillers is performed by a specialized external company. Downtime ~0.5 day / chiller. Yearly maintenance of the 3 CO 2 pumps is done by Nikhef-technicians. Pump is unavailable for >24 hours. Effective downtime of the system: 2 hours / pump. Repair of failing components So far no component had to be replaced, although a (redundant) pressure gauge stopped working in 2012, but miraculously reincarnated after 6 months. Eddy Jans 22
Unexpected problems How do you know a serious problem occured, causing the cooling to go off and the detector to warm up? Especially during LS1. Can t rely on a PVSS-script sending a mail or sms. A modem and sms-routine have been installed in the PLC. antenna modem with Sunrise sim card, so works underground. When a problem occurs every half hour a text message is sent to a list of phones numbers, until the cooling system is again in a proper state. Acted 6 times since Feb. 13 due to failing services. Eddy Jans 23
18 = 8 hours Integrated warm time in 2012: ~1 day 0-10 0-30 year 2012 Eddy Jans 24
Summary cooling system is continuously operational since >4 years, performance is stable and according to specs, redundancy of crucial components has shown to pay off, clogging filters are annoying, good thermal insulation is less simple than it seems, superheated CO 2 can be dealt with, the warning system that sends sms-es is a great tool, thus far the integrated warm time has been ~1 day/year, so. Eddy Jans 25
Outlook lets keep it cool till LS2, when the new VELO pixel detector goes in. Eddy Jans 26