Central DCS Tutorial

Transcription

1 C Tutorial Made by : Etiennette Auffray Helena Malbouisson Andrea Barisone Last Revision : June 27,

2 C Team C Expert Team: Frank Glege (71688, ) Robert Gomez-Reino Garrido (71654, ) Lorenzo Masetti (165820) Giovanni Polese (165807) expert on Call : C Operation: C desk : Etiennette Auffray >> Shift organisation, Tutorial, Twiki : (75844, ) Helena Malbouisson >> Tutorial, Twiki (70861) Andrea Barisone >> Tutorial C shift requirements: CERN ID card and access to P5 and USC ( request done by EDH) All safety trainings up-to-date, including the Biocell training. For more information, please visit: ; At least the token. 2

3 CMS Slice View 3

4 Workstation : Screens Cooling Panel Racks Panel Sensors 4 CMS Panel DSS Screens Main Panel Cavern Access GTPM Panel alarm Panel Subdetectors Panel 4

5 Cavern Access : Screens Cavern Access Displays WP Display Intercom Experimental Cavern Doors Telephone Speaker for the UXC and USC Cavern 5

6 DSS Screens UXC 55 USC 55 6

7 Mobile Phone Mobile Phone 7

8 C General Guideline As a general guideline when in PHYSICS data-taking, all variables controlled and monitored by c should be GREEN (= ready for physics data-taking); If not green, a corresponding alarm will be raised and will appear in the c Alarm Screen; Therefore it is very important to monitor the alarm screen during your shift and have prompt action in case of any alarm; In the next transparencies, the c shifter tasks will be visited one by one, keeping in mind this general guideline. 8

9 Main Tasks 1. Login 2. Which are the common checks and operations according to the different LHC beam modes : Handshake Procedure Pixel and Strips HV Procedure (Concerned Screen : Main Panel) 3. What to do when a alarm occur during the Shift (Concerned Screen : Alarm Panel ) 4. What to do when when a DSS alarm occur during the Shift (Concerned Screen : DSS Alarm Panels) 5. Monitoring Operation: Subdetectors Services (Racks and Cooling) 6. Underground Cavern Access Management (especially in the Technical Stop or Machine Development) 7. Elog : (When and How use it) 8. Safety Tour ( Surface Zone and USC55) 9

10 Login The login is done via your CERN card. Place it on the card reader. You won t be able to do any actions if you are not logged in; Automatically the system should assign you the CENTRAL_ role (if other choices are showed as available due your activities in CMS you have to select manually CENTRAL_ role by yourself ) Please note than when you will perform the Safety Tour you have to take back your CERN card and the Shift Leader should replace you login himself with the same procedure ( this time the system should assign him/her automatically the Shiftleader role) 10

11 Login Verify you are logged in all display: (if not call the expert) 11

12 Alarm Screen This screen should be : Checked regularly Empty Each time an important problem occurs (subdetectors, services, etc ) an alarm (together with an audible alert) should appear on this screen 12

13 Alarm Screen Below you find a row where there are several information regarding the alarm : select and right click on it and a pop up window will appear On the first line of the page you can see which subdetector is affected by the problem : the alarm will continue blinking until handled by the shifter Read Instructions to correctly handle the alert : In case you don t know how to solve it call the responsible If there is no instruction send an to us 13

14 Alarm Screen SUBDETECTORS MAIN LINE: Colors meaning list : LIGHT BLUE : No errors RED: Fatal alert ORANGE: Error YELLOW: Warning GREY: Alert acknowledgeable Direction box : CAME : The condition which triggered the alarm is still present WENT : The condition which triggered the alarm has disappeared Time stamp (when shifter has clicked on Alarm Help) 14

15 Main Panel Subdetectors interface (A) : Control /Modify their status Check their physics readiness Communication interface with LHC (B): Injection or Adjust Handshake Follow the LHC Beam Mode Sequence A When any subdetector is in ERROR, contact the subdetector shifter and/or shift leader.. Event Log (Handshake Warning) 15

16 CMS state. Allowed states: NOT_READY; PHYSICS; STANDBY; ERROR. SUBDETECTORS IN CENTRAL or IN LOCAL In PHYSICS (ON) BRM HV status % of (included) HV channels ON When IN CENTRAL, your name should appear close to each subdetector s partition. If not, call the expert. Select an individual partition to put IN LOCAL or IN CENTRAL. Main Panel In STANDBY Change the state of an individual partition. Name of the expert who s controlling 16 the subdetector

17 Main Panel How to reply to the request of put any detectors in local: If some expert call you, asking for send in local their own subdetector you must be aware that : They should have informed the SL and received his approval; Usually it s not allowed send in local any subdetector from the Injection to the Stable Beam (only during Ramp Down, Cycling and Setup) The Expert must take the control of it within few minutes ( his/her name must appear in the last column in correspondence with the subdetector s name) For the Pixel and Strips Detector: When the beam mode reaches STABLE BEAMS with machine mode of PROTON PHYSICS, the Shift Leader and shifter should check the state of the Pixel and Strips detectors and be sure that they have automatically turned ON. If not, follow the instructions on the next two pages. 17

18 Turning ON the Tracker If after ~1.5 minutes of STABLE BEAMS the Pixel and Strips detectors have NOT turned to 100% ON and/or are stuck in GOING_TO_PHYSICS state: send the GO_TO_PHYSICS command once (to the corresponding partition) if sending the GO_TO_PHYSICS command once does not bring the specific sub-detector ON, the Shift Leader should call the corresponding Pixel and /or Strips DOC. In case they stay in STANDBY, follow the instructions for manually turn ON the Tracker (see next pages) cross checking them with the the Shift Leader Checklist under the 'Beam Mode = Stable Beams' section. Important: the communication between the shifter and Shift Leader is crucial for good cross checking! 18

19 Turning ON the Tracker (Manually) If after ~1.5 minutes of STABLE BEAMS (machine mode of PROTON PHYSICS ) the Pixel and Strips detectors are still in STANDBY the Shift Leader ( with the support of the shifter ) checks if the beam conditions to decide if turn ON them manually or not. These conditions are the same for Pixel and Strips. Here is the check list with the controls that must be performed (the main part): o INJECTION must be inhibited o BKGD1 and BKGD2 must be below 20 o BKGD3 must be below 40 o BEAM MODE must be STABLE BEAMS The complete procedures are available in the usual spot: If any of these conditions are not met call BRM and Pixel Since 2012 if the injection is allowed (the injection inhibit button is released and its green light is on) Pixels and Strip Tracker won t be able to switch on their HV (as additional security measure) When the conditions for turning on the HV are met, the shifter turns ON their HV Channels. (remember : only after the agreement of the Shift Leader) 19

20 Turning the Tracker ON in STABLE BEAMS BKGD (1-3) and STABLE BEAM mode can be verified following this link: 20

21 For each Machine and beam modes, subdetectors may have an automatic action programmed in. The automatic actions can be: o Protection actions: have the PROTECTED label and are always executed in that particular beam and machine mode; o Expert system actions: have the auto label, can be executed if in CENTRAL and will not be executed when in LOCAL. Desired states per subdetector, for each Machine and beam modes. 21

22 The full Action Matrix of all allowed subdetectors states as well as automatic actions is available on cmsonline 22

23 LHC Handshake When LHC is about to inject beam in the machine, it sends a WARNING to CMS via the LHC Handshake panel. 23

24 LHC Handshake Handshake Mode: AUTOMATIC or MANUAL. DIP (*) communication between CMS and LHC. An automatic check of the subdetectors readiness is performed and their states are changed to a safe state depending on the Beam Mode. By clicking on this box you can see the status o Handshake Mode AUTOMATIC: CMS readiness is sent automatically to LHC. o Handshake Mode MANUAL: CMS readiness is sent by pressing the CONFIRM CMS IS READY button. All the lines should have Ok Status in Result column before the Ready command can be sent (*) DIP: Data Interchange Protocol Really important : 1) Never send the CMS readiness command without the agreement of Shift Leader 2) Once that is done check that the Shift Leader pushes the injection inhibit button (the green light must be switched off ) once the LHC has finished injection and goes to Prepare Ramp mode 24

25 Beam abort Injection inhibit button: (Green light off means that the injection is Inhibited) 25

26 Monitor Subdetector 26

27 Subdetector Panel (PLT) If any alerts are activated call Dean Hidas on If (for any reason) Dean is not joinable switch off the PLT. It is not possible for the shifter to switch PLT back on : this will be done by a PLT expert once any problem has been understood and rectified. It should be noted that not all channels (Low and/or High Voltage) will necessarily be on all the time for PLT. When switching PLT off, some alerts may be triggered due to the fast ramp down speed used. As long as the alerts are gone within a short space of time (within a few seconds) they may be acknowledged in the alert screen 27

28 Subdetector Panel (Tracker) Verify that heartbeats are blinking Check the majority table Check the dewpoints Contact Phone : Pixel : Strip :

29 Subdetector Panel (ECAL ) To operate ECAL partitions, the user has to have their control and be logged in as CENTRAL_ ALL padlocks associated to the ECAL partitions, should ALWAYS be GREEN without any color around it!!! The Supervisor tab has to be ALWAYS selected!!! As shown, ALL objects should ALWAYS be GREEN!!! The EB/EE PLCs Heartbeats indications should change size every ~5 seconds!!! Colour code for all objects from the main window is available 24h support details are provided Last updated 12-Mar-2010 If any of the conditions above is absent, please contact the CMS ECAL expert IMMEDIATELY: 72606!!! 29

30 Subdetector Panel (HCAL ) If anything in error (red) Contact Phone :

31 Subdetector Panel (DT ) Select the DT subsystem whose status is shown by the chamber colors in the wheel widgets. When the refresh is finished this label will change to the selected subsystem (General is an OR of the others ) 6) 1) The bar on top of each wheel blinks if >=1 chamber has communication problems. When the problem is solved, you need to push the Reset button to stop the blinking 5) CLICK ON A CHAMBER TO OPEN ITS PANEL Error conditions flagged by Red color on tabs: 1) Cooling 2) UX / USC Turbines, Smoke alarms 3) MiniCrate Temperatures Error 4) Water Leaks 5) LV Temperatures 6) Gas System 7)>=1 MiniCrate OFF Call except for 6 For 6 call only from 7h-23h 7) 2) 3) 4) Contact Phone :

32 Subdetector Panel (RPC ) Global FSM states for barrel, endcap, hardware, and gas system Panic buttons Percentage of hardware ok, with trend to spot changes over the shift + Majority 9 error conditions flagged by red led on critic subsystems General Detector View List of known problems to be xchecked in case of alarm Global performances barrel + Endcap Contact Phone : In case of error condition, the information about the problem, the time stamp, the details and the number to call is displayed

33 Subdetector Panel (CSC ) Key Login Panel Who to Contact: CSC End View CSC Shifter - shifter address problem CSC CEO shifter contact CEO * CSC * PCrates CSC Shifter Communication Status CSC Shifter Subsystem Panel HV LV FEDs CSC CEO Value Monitor CSC CEO 33

34 Monitor Services 34

35 Service Screens (Rack Panel ) Racks summary Racks Cooling Station. If you click on 1 rack you get details on it Rack Power Status per underground area S1-S4 = Service Cavern X0-X5 = Exp. Cavern Color code: Grey Not used Green OK Blue OFF -> Red ERROR -> requires intervention! Take actions on racks. Follow the instructions 35

36 Service Screens (Rack Panel ) Rack details 36

37 TOP PART OF THE SCHEME Service Screens (Cooling) Cooling Panel Outside During working hours: (8h30/17h30) no matter which TOP doubt PART or change of the screen: of color: CMS cooling support, ORANGE, write on e-log, RED - Call CCC BOTTOM PART of the screen (flow & tank levels) : ORANGE : 17h30/24h00 call h00/7h00 ignore, RED, call Color code: Green OK Yellow warning Orange Error Red FATAL ERROR Blinking Values requires intervention! BOTTOM PART OF THE SCHEME Copper circuit leak Warning: 1 l/h > 60 min Alarm: 4 l/h >30 min or 8 l/h >1 min Aluminum circuit leak Warning: 1 l/h > 60 min Alarm: 4 l/h >30 min min or 8 l/h >1 min Rack circuit leak Warning: 2 l/h > 60 min Alarm: 6 l/h >30 min min or 8 l/h >1 min

38 Service Screens (Cooling) Blinking values If a value is blinking: you should verify at if there are invalid CMS cooling values. In case the bullet corresponding to CMS Cooling is red, it is very likely to be a problem on the CCC side. You should call the CCC (72201) to inform them there is a problem with their publications and point to them what you see from the link above. If there are no invalid values in the link above, call the c expert and inform about the problem. The interface from above is slow, so you should wait about 10 minutes to make sure it is updating, before reporting the problem. 38

39 GTPM : Gestion Technique des Problèmes Majeurs For expert only : Same panel as in CCC. 39

40 Sensor 4 CMS The Sensors 4 CMS groups geographically and makes available existing sensor data from CMS, publically available from different subdetectors or technical support groups. It is organized in regions of CMS and shows information on temperature and humidity. Detailed information in this twiki: 40

41 DSS :Detector Safety System In case of alarms: read the alarm description and contact the responsible person (e.g. Magnet, subdetectors, etc.); In case the alarm description is not clear: during working hours: call DSS expert, Alain Meynet: ; out of working hours: call Christoph Schaefer: or Wolfram Zeuner: and at last (in case the other numbers failed). Check the DSS display is Alive: These buttons should blink. If not, restart the panel (instructions in the DSS binder). If restart does not work, call :

42 Front End warning light : should be always green. If not, call Alain Meynet, between 7:00 and 22:00. If red during the night, call Alain Meynet in following the morning. DSS Alive. Should blink. User should always be set to user. If not, click on Logout. The panel will restart automatically. DSS :Detector Safety System If there is a DSS Alarm with Siren and light, the shifter should immediately call the Technical coordination number Stop Siren button: stops the siren in the DSS hardware panel when there is an alarm. Only works from the DSS USC55 panel! Right-click on the alarm and select Show to obtain more information. If the alarm s Triggered state is red, it cannot be acknowledged. To acknowledge an alarm: 1. click on the ALARM column, on the DSS ALARMS list; 2. Then click on the ACTIONS column of the DSS ACTIONS list.

43 DSS :Detector Safety System DSS Hardware Piquet:

44 Elog ELOG address : cmsonline > Common > Elog > Subsystems > At the beginning the shift: write an elog with the summary status of all the c systems; During your shift : Write only a short elog with specific information when you have any problem with subdetectors, cooling, racks, etc... with indication on your elog Title of the problem At the end of the shift: Write an elog with a summary of all the relevant events that happened during your shift. Shift Instructions are available online at: structions 44

45 Before Every Shift It is recommended to: o read the previous ELOG entries to get familiar with the latest c events; o Read the c twiki; o If you are an experienced shifter, look for new or updated items in c News: o Read the c shift tutorial. 45

46 Safety Tour Tour during the working days is done by CMS technician, the shifter has to do only the tour during night shift and the non working days ( in these case the tours must be done once a shift, if the shift is not too busy) The shifter must wear safety shoes and helmet with headlamp. The shifter must bring with him/her a mobile phone and fill in the paper version of the 'Access to Experimental and Service Caverns Let the shift leader know if you leave your desk, he has to put his CERN badge on the card reader and check the displays during your absence. Before their first shift newcomers should do a training tour (usually just after the tutorial); Write a slimos elog (cmsonline > Common > Elog > General > Slimos) in the text separate new problems from old one tour instructions are available on the c desk and online at: erary 46

47 Safety Tour : Surface Zone Surface Point Doors of the SX5 are locked during weekends and at night. Hall is accessible only through this door which are equipped with the card reader 3 CMS Safety Website: 47

48 Safety Tour : USC55 S2 3 EN/CV area 2 Magnet Area 4 5 Down Underground Control Room S1 Pump room 1 up to S

49 Underground Cavern Access Every person going underground (Service and Experimental caverns) is required to fill in the paper version of the 'Access to Experimental and Service Caverns' list that can be found in a binder by the Shift Leader's desk; The c shifter has to check that the person fills this form when he/she goes into the cavern and when he/she returns; The person has to have a mobile phone; The person has to inform to you or the shift leader the reason to go to the cavern; If after some time the person is not back in the control room, call him/her to check everything is ok. You can find detailed information on underground access in the c twiki: 49

50 Cavern Access : Screens Cavern Access Displays WP Display Intercom Experimental Cavern Doors Telephone Speaker for the UXC and USC Cavern 50

51 Underground Cavern Access 1. Each person that wants to access the experimental cavern has to be properly registered in a WP; 2) The Access management software Work Packages (WP) is reachable from the CMS Online Page ( clicking on ACT ) or directly from this url : page/portal/cms%20online%20s ystem/act

52 Underground Cavern Access 3) Once inside Shifter has to select : Document Database

53 Underground Cavern Access 4. go up on the left side of the page and select Shifter Page ) 4 53

54 Underground Cavern Access 5. Enter the name of the person willing to enter of the cavern 6. The list of assigned WP will appear 7. Select the WP to see its status. 8. To start the WP Select in the menu status : Ongoing 5 6,

55 Underground Cavern Access 9. Turn the dosimeter ON; (Push the black button three times : once for each message will pop up on its little screen (CHANGE >> ENTER >> 1mv) : then a red light will flash few seconds and also you will hear a bip ) 10. Select the dosimeter number in the Operational Dosimeter field; 11. Read the initial dose from the dosimeter and fill it in the appropriate field of the Shift Operation page; 12. Click Ok to grant access to the person; Point 10 to By the return of the dosimeter, fill in the final dose and click Ok ; 14. Turn the dosimeter OFF (simply push the black button on top of the dosimeter, the word CHANGE will appear, keep on holding the black button until the word GO OFF shows up. Then immediately release the black button. If one waits for too long to release the black button, the 55 dosimeter will not switch off).

56 Underground Cavern Access 9-14 Button Dosimeter number 9. Turn the dosimeter ON; (to do that, push the black button three times : once for each message will pop up on its little screen (CHANGE >> ENTER >> 1mv) : then a red light will start flashing) 10. Select the dosimeter number in the Operational Dosimeter field; 11. Read the initial dose from the dosimeter and fill it in the appropriate field of the Shift Operation page; 12. Click Ok to grant access to the person; 13. By the return of the dosimeter, fill in the final dose and click Ok ; Red flashing light Dosimeter display After an access: 9-14 shifter has to make sure the lights in UXC are switched OFF. 14. Turn the dosimeter OFF (simply push the black button on top of the dosimeter, the word CHANGE will appear, keep on holding the black button until the word GO OFF shows up. Then immediately release the black button. If one waits for too long to release the black button, the 56 dosimeter will not switch off).

57 Underground Cavern Access 15. Click on edit button 16. The WP summary page will appear 17. In the list of name click on edit

58 Underground Cavern Access 18. Insert the name of the person you want to register (previously authorized by the WP leader) 19. Select his role 20. save changes 21. The new name will appear in the name list

59 Underground Cavern Access Once the authorized person has received his dosimeter (s)he can finally access to the cavern, and in the two displays here below the Shifter can check all the cameras pointing at the different doors in the cavern : When the person reach the entrance (21) there are still few steps to be execute 21

60 Underground Cavern Access 22 The name will appear here. 23 he needs to pass his badge over the card reader in the yellow underground cabinet (22) : Then on the screen his name will appear in the dedicated window (23 and the Shifter has first to select the line and then click the button Give the Key (24) in order to let the drawer in the underground rack opening (25) and delivering the access key to him Once selected the name the button will became clickable

61 Useful Contacts (1) Emergency, accident, fire : Any technical problem (Siren Alarm): Rack Problem : Rack becomes off or in error : 1. Tracker (Pixel) : Tracker (Strip) : ECAL : or HCAL : , 5. CSC : DT : RPC : During the day you can also call Silverio : Cooling Problem : 1. ECAL cooling: 70130/ One (other the ECAL) of the box is not green : During the day : Cooling piquet Outside working hours : Top part : call CCC Bottom part : call if red Screen blinking : Call CCC Magnet Problem : Call Magnet piquet Cryogenic problem : Call Cryogenic piquet Gaz Problem : Call Gaz Piquet Problem: C :Call expert on call Subdetector : 1. Tracker (Pixel) : Tracker (Strip) : ECAL :72606 or HCAL : , 5. CSC : DT : RPC : DSS Problem : Alarm : Alain Meynet during the day Christoph schaefer Technical Problem : Hardware Problem :

62 Useful Contacts (2) TS Technical control room (CCC): IN CASE OF ANY TECHNICAL PROBLEM CALL: Backup: Austin BALL: , Christoph SCHAEFER:160202, Wolfram ZEUNER: , Martin GASTAL: expert on call : Backup : Frank : , Bobby : , Lorenzo : DSS expert on call : CASTOR: TOTEM (Green Barrack): BRM/BCM: Fireman at P5 (working hours ) : (still call directly the in case of DSS alarms) Online Web Contact List :