Distribution Best Practices Program 2012 Gas Control Room Management Best Practices Roundtable Group 2 Best Practices Identified to Address Challenges *Notice: This document is based on an informal Best Practices Roundtable and is for general information only. Responses are not intended to bind any company or state a company's official position. The information represents an unaudited compilation of information and could contain coding or processing errors. AGA makes no representations about the accuracy of the information in the publication or its appropriateness for any given purpose or situation. Copyright & Distribution: Copyright 2012 American Gas Association. All rights reserved. This work may not be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by information storage and retrieval system without permission in writing from the Southern Gas Association American Gas Association.
Topic 1 Alarm Management Challenges Identified How do you actually manage the process of Alarm Management? How are companies defining Safety Related Alarms? When there is field work going on that brings system/pipeline outside of normal operating conditions, how do handle alarms? Do we delete existing alarms and create new alarms based on field conditions? If we do not delete alarm conditions used under normal conditions and they go into alarm, would these alarms count against the maximum allowable alarms per hour? What is the best process to introduce new screens/points into the system; especially when the screens/points have not been completely field tested? Integrating separate alarming systems into one reporting and management process (SCADA alarms, electronic recorder alarms, auto-dialers, etc.) a) Do you combine them into one reporting process, keep them separate, or only report SCADA Should Alarm Management cover more than Safety Related Alarms? Should we go as detailed as ANSI/ISA 18.2 2009 or RP-1167 in our alarm management plan? How are companies dealing with the level of authority given to controllers to change alarm limits? Page 1 of 11
Alarm Management Challenge #1: How do you actually manage the process of Alarm Management? a. Have monthly alarm review where points are reviewed. Still trying to find the best way to define a false alarm. Challenge it s a manual process. When they transition to the newer technology by the following year, process should be improved. Newer system will streamline their process and give them more capabilities. R. Vallejo, Southwest Gas b. Came up with a hard and fast rule. M. Hall, NiSource i. Identified the number of times that an alarm went out of the normal operating range. (Benefit: Identification) ii. A nuisance alarm occurs more than 6 times in 30 minutes. (Benefit: Frequency) iii. Breakdown weekly report into acceptable and nuisance alarms. (Benefit: iv. Reporting) Follow-up with field staff to find out how the alarms were handled. Getting the alarms out to the field in a timely fashion. Reinforces feedback to the controllers that you are taking action on the alarms. (Benefit: Follow-up/QC) c. Process team operates gas network. Process team looks at it quarterly. K. Wirth, Enbridge Gas d. Uses a system call Wonderware for SCADA. iwaf (integrated Web Application Framework) is embedded in Wonderware, so controllers can look at the report per shift and they look at high-high only. The controllers enter their comments into SCADA and manager reviews false. T. Shuck, Atmos Energy e. Use Excel and pivot reports to sort alarms and determine high alarms. Very time consuming. Able to identify how many different alarms you have and how often they were occurring. D. Gladstone, DTE Energy MichCon f. Schedule staff sends report to manager and it s reviewed. If a high number of alarms appear, that indicates there s an issue. J. Martin, CenterPoint Challenge #2: How are companies defining Safety Related Alarms? a. Engineering and Gas Control sit down and review all system points and determines if it affects safety or reliability. Being able to provide customers with safe and reliable service. T. Nodes, Xcel Energy b. Kansas Gas does not define communication failures as safety related. Communication failures is an everyday occurrence so not including communication failures ensures that they are only getting safety related alarms. S. Luthye, Kansas Gas, B. Hoffman, UGI Utilities c. Sat down with the operations folks to identify what is a call-out or safety-related point. Highs and lows are alerts. High-highs (MAOP) and low-lows (contractor deliverability) are alarms. M. Hall, NiSource d. Have rate of change (ROC) alarms. A. Redding, Consumers Energy e. Have them on distribution system for cut line. Quick, early notification. K. Blackmon, Alagasco, B. Hoffman, UGI Utilities Page 2 of 11
f. They use rate of change alarms on medium and high pressure systems as a warning for operators that a system change occurred. ROC alarms provide a means to alert operators to potential line issues or ruptures. One issue can be that if alarm limits are set close you may only get one ROC before a true alarm comes in. B. Hoffman, UGI Utilities g. Have three levels of alarms. Anything deemed an alarm, i.e. any of those 3 levels of alarms, is a safety-related alarm. S. Luthye, Kansas Gas h. Use a matrix of impacts vs. severity levels and if consequences of an unmanaged alarm could result in significant human or environmental impact (i.e., high level severity), then it is considered a safety-related alarm. This is built into their alarm rationalization process. It makes the alarm rationalization process less subjective with defined impacts and severity levels. A. Millerick, Integrys Challenge #3: When there is field work going on that brings system/pipeline outside of normal operating conditions, how do you handle alarms? Do you delete existing alarms and create new alarms based on field conditions? If you do not inhibit alarm conditions used under normal conditions and they go into alarm, would these alarms count against the maximum allowable alarms per hour? a. If we know someone is going out in the field to do some work, then the alarm is inhibited. Do point to point verification on SCADA. Uses SCADA Lotus Notes. Perform MOC verification and everybody gets copied on it. Doing this for any calibration that takes 15- minutes average. They never delete existing alarms. M. Hall, NiSource b. With alarms that come into SCADA with engineering projects, they will change the limits to be able to track the procedure and ensure that any changes from the procedure will be captured by the system and alert the controller of a problem. If it s construction, e.g. procedure for tie-in, they ll drop the alarm limit. T. Nodes, Xcel Energy Challenge #4: Integrating separate alarming systems into one reporting and management process (SCADA alarms, electronic recorder alarms, auto-dialers, etc. Do you combine them into one reporting process, keep them separate, or only report SCADA? a. ERX. Two controllers on shift, one looks at ERX, not in SCADA. Developed through Honeywell. Basically an electronic chart recorder. Pressure alarms on them. K. Wirth, Enbridge Gas b. Mini s and ERX s. Both come into SCADA. T. Nodes, Xcel Energy c. Using ERX. Use SCADA engine to poll it. Looking at replacing charts with ERX system. M. Hall, NiSource d. In wintertime, especially if it is critical, will call then. Monitoring all of them in the Control Room. Have alarms on the pressure indicators, if it is critical. If you get an alarm, duty Page 3 of 11
calls. If an alarm goes off, call tech who goes out to handle the alarm. T. Baker, CenterPoint Energy e. Will continue to keep calling. T. Nodes, Xcel Energy f. Targeting 1 to 2 ERX per system to align with the gas for intermittent weather conditions. 1,600 to 1,700 in the next 6 years. Visibility strategy. Want to have continuous polling. Battery operated once a day. K. Wirth, Enbridge Gas Page 4 of 11
Topic 2 Change Management Challenges Identified How are companies with multiple utilities within the corporate family who use a single, consolidated Control Center integrating the change management requirements of CRM? What is the best way to encourage outside groups to notify the control room when field work is complete so that normal operating parameters (alarm configurations, for example) could be restored and change request ticket could be closed out? How do we handle non-cooperating groups? How are companies determining the best way to document changes, big or small? What level of documentation is enough? For which changes made to field devices (e.g. operating settings, control parameters, transmitter calibrations, etc.) do pipeline operators require what level of Gas Control review and approval Who handles change management process in gas control? What technology is used for change management in gas control? Page 5 of 11
Change Management Challenge #1: How are companies with multiple utilities within the corporate family who use a single, consolidated Control Center integrating the change management requirements of CRM? a. Told all centers to call gas control and communicate; easier than telling everyone in the field about all telemetry points. If there is no effect in the control center, then it only gets logged, not change management. T. Baker, CenterPoint b. Management of change is considered a planned change, form sent out and multiple groups sign off on it. It is a corporate change process, not just gas control. Management of change should get incorporated into corporate standards. M. Hall, NiSource Challenge #2: What is the best way to encourage outside groups to notify the control room when field work is complete so that normal operating parameters (alarm configurations, for example) could be restored and change request ticket could be closed out? How do we handle non-cooperating groups? a. Went into Union offices and explained new CRM rules. Received well. A. Redding, Consumers b. Vital to bring corporate into the changes that are occurring within the control room. M. Hall, NiSource c. Gas Control order that must be called in. Asked question about how many companies have outage coordinators, several companies do. T. Baker, CenterPoint d. Developing entire department dedicated to outage coordination. K. Wirth, Enbridge Gas Challenge #3: How are companies determining the best way to document changes, big or small? What level of documentation is enough? a. Big Change = Management of Change. Small Change = SCADA Lotus Log. All logs can be chosen through one interface in Lotus Notes. Most will be in the SCADA log. M. Hall, NiSource b. Written procedure for plate change, put into station notes. A. Redding, Consumers c. If it goes past the log, then it goes into SharePoint and is considered change management. T. Baker, CenterPoint d. Everything is embedded into SCADA system. Logging done through same system as SCADA. Wonderware used as HMI. T. Shuck, Atmos Energy Page 6 of 11
Challenge #4: For which changes made to field devices (e.g. operating settings, control parameters, transmitter calibrations, etc.) do pipeline operators require what level of Gas Control review and approval. a. Any device that they can see within SCADA. They call and conduct a point-to-point. Management of change process is done. Review and approval must be done also. There is a formal process for operating settings. M. Hall, NiSource b. A review is performed but approval is predetermined by system operations. T. Baker, CenterPoint c. Documentation sent in, and must be approved before any work has taken place. J. Martin, CenterPoint d. A person gets settings from system planning, makes any adjustments necessary and sends them out to field operations for implementation. A. Redding, Consumers Energy Challenge #5: Who handles change management process in gas control? a. CRM specialist drives process from gas control. M. Hall, NiSource b. Have a trainer that manages the changes. Put it into shift change log and hard copy, must log off before signing into SCADA. K. Wirth, Enbridge Gas c. Have a trainer. B. Andino, PG&E, A. Redding, Consumers Energy d. Managers/Lead Controllers. CenterPoint LDC e. Have a specialist. T. Baker, CenterPoint f. Managers. Kansas Gas, UGI Utilities, AGL Resources, Kansas Gas Challenge #6: What technology is used for change management in gas control? a. SharePoint. K. Wirth, Enbridge Gas b. Lotus Notes. M. Hall, NiSource c. Maximo (IBM), passed inspection. S. Luthye, Kansas Gas d. SAP, allows signatures. PG&E e. APEX. UGI Utilities Page 7 of 11
Challenges Identified Operations Best Practices Program Roundtable Notes Topic 3 Adequate Information What are companies doing to educate their controllers on remote systems? How are companies integrating point-to-point verification into their change management process? Are companies requiring an on-duty controller to be directly involved in the point-to-point verification tests or simply a member of the control room staff? How do companies track or monitor whether controllers are correctly categorizing events (AOC, EOC, etc.) as they happen? For point-to-point verification, they would like to know for which changes made to field devices (e.g. software downloads, like-for-like replacement, etc.) do pipeline operators require what level of verification/documentation. What are companies doing to educate their controllers on remote systems? Challenge #1: What are companies doing to educate their controllers on remote systems? a. Send controller to field to learn what s happening. They take pictures while they re there for documentation. They have a network drive to upload the pictures. The pictures are tagged with description. Everyone can see what they did while they were on the field trip. They do a review of the field trip so the other controllers can see what s going on in the field. M. Hall, NiSource b. We have buy-in from field managers. Field managers will come in to the Control Room and spend time with the controllers. K. Wirth, Enbridge c. Create group screens. T. Baker, CenterPoint d. Information is interjected in the market write-up. M. Hall, NiSource Challenge #2: How are companies integrating point-to-point verification into their change management process? a. Change management form includes point-point verification. M. Hall, NiSource b. MOC form with check box if P-P is required. Full P-P done if changes to RTU, SCADA. Meetings set up with measurement specialists and Gas Control. CenterPoint Transmission, Xcel Energy c. 2 control rooms on one SCADA system. Will have a test call before a cutover. T. Nodes, Xcel Energy d. Field Personnel can do P-P and coordinate with control room. R. Vallejo, Southwest Gas Page 8 of 11
e. Form held with Field and Ops personnel. CenterPoint LDC f. Standardized form in SharePoint, re-commission P-P back into SCADA. K. Wirth, Enbridge g. Only doing point-point changes on new installations or changes. S. Luthye, Kansas Gas Challenge #3: Are companies requiring an on-duty controller to be directly involved in the point-to-point verification tests or simply a member of the control room staff? a. Neither, SCADA staff does P-P. Kansas Gas, UGI Utilities, PG&E b. Gas Controllers. M. Hall, NiSource c. Controllers, but not necessarily on-duty controllers. CenterPoint LDC d. Qualified specialists. CenterPoint Transmission Challenge #4: How do companies track or monitor whether controllers are correctly categorizing events (AOC, EOC, etc.) as they happen? a. There is a difference between Abnormal Operation and Abnormal Operating Condition. An AOC may only be related to a technician. Volume Specific. Each state is different. We do it for both transmission and gathering systems. J. Martin, CenterPoint Transmission b. Document the occurrence. See if controller fully documented the situation. Does not necessarily have them categorize the situation. M. Hall, NiSource c. Every AOC documented in system has to be closed. Controllers make that decision. S. Luthye, Kansas Gas d. Operator log is categorized and broken down into AOC or EOC categories. Log entry you can choose and/or change category. Only related to Gas Control. W. Rutledge, AGL Resources Challenge #5: For point-to-point verification, they would like to know for changes made to field devices (e.g. software downloads, like-for-like replacement, etc.) do pipeline operators require what level of verification/documentation? Any change to field devices will require P-P. NiSource, CenterPoint Transmission Partial, whatever is affected. Atmos Energy, Southwest Gas, CenterPoint LDC, Enbridge Note: No P-P required for communication link Page 9 of 11
Open Forum Discussion: Audit Overview (NiSource) Checked manager work schedule. Wanted travel times for each operator (used Google maps). Time 14 hours total; 1hr maximum driving. Have hard copies of CRM plan ready several. Must show plan if SCADA shuts down. Did not talk to controllers. Wanted to see how Management of Change process works (re-audit in the future). Ohio Audit. Audit Overview (Kansas Gas) Only under State Jurisdiction. Invited PHMSA to participate, more detail oriented and drove audit. 3 months warning. Document was sent ahead of time. Full 56-page PHMSA audit. 3 days length. Separate conference room setup. Recommended. They wanted a procedure for everything. They want to see two columns, field values and SCADA values, should be visible. Third column also asked for, P-P field, SCADA and how the technician left the point, i.e. an as-found. However this last column is not actually asked for. AOCs that occur simultaneously and in sequence. Training about AOC was asked for. Document processes and procedures. They asked for everything. Including startup and shutdown of backup control center. They wanted procedures for impromptu breaks, i.e. less than 30 mins. Comprehensive audit would only be done every 5 years. There would be a visit yearly but it would be superficial. Page 10 of 11
Have only the minimum required of Gas Controllers needed. They will interview everyone. Audit Overview (SourceGas) They have 2 control rooms. Needed to justify what would require the SCADA system to be switched from one control room to the other, e.g. a fire drill. Wanted to make sure Managers are not going over work time limits. Mangers have to create their own documentation of work hours. Happy with their shift-change procedure. Wanted several copies of CRM in place. Page 11 of 11