Alarm Management Plan

John E. Bogdan, Susan F. Booth, & David P. Garcia Abstract Why prepare an Alarm Management Plan just to meet PHMSA requirements (49 CFR Parts 192.631 and 195.446) when, for essentially the same effort, you can prepare one that, when implemented, will improve your bottom line? This paper has three objectives: 1. Explain the fundamentals of constructing an alarm management plan to meet PHMSA requirements including personnel requirements, scope, content, and maintenance; 2. Examine using in-house resources versus an external consultant; 3. Encourage operators to prepare an alarm management plan that not only will satisfy PHMSA requirements, but also will improve safety, productivity, and profitability when implemented. copyright J Bogdan Consulting LLC 2010 All Rights reserved

Table of Contents Introduction... 3 Definitions... 3 1 Constructing an alarm management plan... 3 1.1 Get educated... 4 1.2 Capture knowledge of your operation/organization... 4 1.2.1 Core values... 4 1.2.2 Roles and responsibilities... 5 1.2.3 Alarm system design guidelines... 5 1.2.4 Alarm system operation, maintenance, and review guidelines... 5 1.3 Organize your information and align it with current standards and regulations... 5 1.4 Write the alarm management plan... 9 2 In-house development versus outsourcing... 9 3 Thinking beyond the minimum... 10 Conclusion... 11 2 copyright J Bogdan Consulting LLC 2010 All Rights reserved

Introduction On December 3, 2009, the Pipeline and Hazardous Materials Safety Administration (PHMSA) amended the federal pipeline safety regulations to address human factors and other aspects of control room management (49 CFR Parts 192.631 and 195.446). In part, these regulations require pipeline operators to have a written alarm management plan by August 1, 2011 and to have implemented that plan by February 1, 2013. This paper: 1. Discusses how to construct an alarm management plan to meet that requirement; 2. Examines developing the plan using only in-house resources versus augmenting them with an external consultant; 3. Encourages pipeline operators to use the implemented plan as a vehicle for continuous improvement, resulting in a better bottom line. Definitions An alarm is a visible and/or audible means of indicating to the controller an equipment malfunction, process deviation, or other condition requiring a controller s response. It is generated by a process variable crossing a defined threshold into an undesirable or hazardous region. An alarm system is the collection of hardware and software that detects an alarm state, communicates an indication of that alarm state to the controller, and records changes in the alarm state. An alarm management plan is a written framework that defines how the alarm system is to be designed, operated, maintained, and reviewed. 1 Constructing an alarm management plan It is recommended that you designate an alarm management coordinator and an alarm management team to oversee construction of the plan. The coordinator should have knowledge of both the technical and operations facets of your organization. The team should be comprised of personnel from all levels - up, down, and across the organization to help ensure that all the relevant disciplines and viewpoints are considered and to increase acceptance of the alarm management plan. 3 copyright J Bogdan Consulting LLC 2010 All Rights reserved

The construction of an alarm management plan can be broken down into the following steps: 1. Get educated on the principles of alarm management and current guidelines, standards, and regulations. 2. Capture knowledge of your organization/operation. 2.1. Identify core values. 2.2. Define roles and responsibilities. 2.3. Determine desired alarm system design guidelines. 2.4. Determine desired alarm system operation, maintenance, and review guidelines. 3. Organize your information and align it with current standards and regulations. 4. Write, edit, and review your alarm management plan. 1.1 Get educated Many personnel, especially those on the alarm management team, will need to be educated in varying degrees, and the alarm management coordinator will need to acquire broad knowledge of alarm management, including basic principles, best practices, available technology, and industry standards and regulations. Information can be obtained through reading books and articles, reviewing industry guidelines and standards, and/or attending training classes or seminars. Table 1 contains some recommended starting places. Table 1 Suggested Educational Resources Books Standards Training Rothenberg, Douglas H. Alarm Management for Process Control. 1st Edition. New York: Momentum Press, LLC, 2009. API RP1167 - Recommended Practice for Alarm Management DRAFT Apr. 15, 2010 TIPS, Inc. ANSI/ISA 18.2 2009, Management of Alarm Systems for the Process Industries ISA, Introduction to the Management of Alarm Systems (IC39C) 1.2 Capture knowledge of your operation/organization After alarm management fundamentals are mastered, the coordinator and team, possibly with the help of an expert, will study the current state of the alarm system and determine what elements are in place and what is needed for an effective system in line with regulatory requirements. They will extract from the organization the information necessary to write the plan. Major elements are: 1.2.1 Core values Alarm systems are designed to help manage undesirable situations. To understand what is undesirable for you, it is essential to first define your organization s core values. For example, key values for most organizations are profitability, safety, and environmental responsibility, and there may be others such as positive community relations. Because all undesirable situations do not impact each core value and do not have the same potential severity of consequences, it is necessary to establish relative levels of severity within each core value. This analysis is the basis for prioritizing and classifying alarms. 4 copyright J Bogdan Consulting LLC 2010 All Rights reserved

1.2.2 Roles and responsibilities The next step is to clearly define roles and responsibilities for all those who use and support the alarm system. For example: Who will maintain the alarm management plan? Who will communicate it to the organization? Who will maintain the alarm system? Who will review it? 1.2.3 Alarm system design guidelines Essential requirements for accepted alarms are determined and drawn up, creating a guideline for detailed design. Some typical questions addressed are: How will set points be calculated so that the alarms are presented to the controller with appropriate time to respond? How will alarms be presented to the controller? What style elements are important (e.g. sound/tone, color, and blink)? What kind of alarms does the SCADA system support? Which ones will be used? How will they be used? What advanced alarming techniques will be used? How will they be used? 1.2.4 Alarm system operation, maintenance, and review guidelines Issues surrounding the operation, maintenance, and review of the alarm system are considered, and guidelines for normal operations, training, upkeep, testing, repair, updating, and auditing of the system are prepared. Some typical questions addressed are: Are there any special training requirements for certain classes of alarms? What alarm system metrics should be measured? What are the action limits? To whom should deviations be reported? How frequently should the alarm management be audited? What is the scope of the audit? How should changes to alarms or the alarm system be managed? 1.3 Organize your information and align it with current standards and regulations Once you have completed the extraction of information from your organization, you will need to organize it into a useful form and verify that you have satisfied any applicable regulations. Table 2 below should be useful in aligning information with PHMSA alarm management requirements (49 CFR Parts 192.631 and 195.446). Column 1 is the PHMSA regulation. Column 2 describes the tasks required to comply with a specific requirement in the regulation. Columns 3 and 4 list the relevant sections in the AGA Alarm Management for Control Room Operations in the Natural Gas Industry and ANSI/ISA 18.2 2009 Management of Alarm Systems for the Process Industries documents respectively that provide guidance for the required tasks. 5 copyright J Bogdan Consulting LLC 2010 All Rights reserved

Table 2 - PHMSA Alarm Management Requirements PHMSA Requirement (e) Alarm management. Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms. An operator s plan must include provisions to: (1) Review SCADA safetyrelated alarm operations using a process that ensures alarms are accurate and support safe pipeline operations; Required Tasks to Comply Select and educate an alarm management team, define core values, assign roles and responsibilities, gather and organize relevant information, and write the alarm management plan. Develop and implement a review process to ensure that undesirable events are correctly identified and properly alarmed (i.e., alarms must be prioritized and presented to the controller in a clear and timely fashion). AGA Alarm Management for Control Room Operations in the Natural Gas Industry Alarm Philosophy Alarm Determination Roles & Responsibilities - Document the analysis & action ANSI/ISA-18.2-2009 Management of Alarm Systems for the Process Industries Alarm Philosophy** Identification Rationalization Design 6 copyright J Bogdan Consulting LLC 2010 All Rights reserved

Table 2 - PHMSA Alarm Management Requirements PHMSA Requirement (2) Identify at least once each calendar month points affecting safety that have been taken off scan in the SCADA host, have had alarms inhibited, generated false alarms, or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities; (3) Verify the correct safetyrelated alarm set-point values and alarm descriptions [when associated field instruments are calibrated or changed and]* at least once each calendar year, but at intervals not to exceed 15 months; Required Tasks to Comply Develop a monitoring program to identify alarms that are generating false alarms and alarms that have been set to not annunciate (e.g., disabled, inhibited, suppressed, shelved, out-of-service, off scan, forced, manual). Develop a master list or database of all relevant alarms and their key parameters, and compare that master list to the set of installed alarms to identify and resolve any unauthorized changes. AGA Alarm Management for Control Room Operations in the Natural Gas Industry - Performance metrics Alarm Resolution Management of Change ANSI/ISA-18.2-2009 Management of Alarm Systems for the Process Industries Monitoring & Assessment Management of Change Maintenance - Periodic testing 7 copyright J Bogdan Consulting LLC 2010 All Rights reserved

Table 2 - PHMSA Alarm Management Requirements PHMSA Requirement (4) Review the alarm management plan required by this paragraph at least once each calendar year, but at intervals not exceeding 15 months, to determine the effectiveness of the plan; (5) Monitor the content and volume of general activity being directed to and required of each controller at least once each calendar year, but at intervals not to exceed 15 months, that will assure controllers have sufficient time to analyze and react to incoming alarms; and (6) Address deficiencies identified through the implementation of paragraphs (e)(1) through (e)(5) of this section. Required Tasks to Comply Conduct an annual audit of the alarm management program by comparing managerial and work practices to procedures, procedures to policy, and policy to regulations and industry guidelines. Measure the alarm system performance and compare it to established targets. Typical measurements would include alarm rate, alarm flood (size and duration), chattering/fleeting, duplicate, false, frequent, inhibited/forced/manual, off-scan, standing/stale, and unauthorized changes. Develop and implement any action plans or recommendations identified as a result of programs developed in steps 1 through 5. AGA Alarm Management for Control Room Operations in the Natural Gas Industry - Performance metrics ANSI/ISA-18.2-2009 Management of Alarm Systems for the Process Industries Audit Monitoring & Assessment Audit Others - Design - Implementation - Maintenance - Management of change *The requirements in [] are specific to Part 195 Transportation of Hazardous Liquids by Pipeline. ** The term Alarm Philosophy as used by ANSI/ISA-18.2 is synonymous with the term Alarm Management Plan as used in the PHMSA ruling. 8 copyright J Bogdan Consulting LLC 2010 All Rights reserved

1.4 Write the alarm management plan Finally, a draft of the plan is prepared and sent throughout the organization for review. The draft is edited and reviewed as necessary until a satisfactory version is produced and approved. Periodic review of the approved plan is necessary and should be carried out as specified in the plan. Table 3 below is a sample table of contents of an alarm management plan. Table 3 - Alarm Management Plan Table of Contents 1. General 1.1. Purpose of alarm management plan 1.2. Purpose of alarm system 1.3. Definitions 1.4. Related site procedures 1.5. References 1.6. Responsibilities and roles 2. Identification 3. Rationalization 3.1. Rationalization Team 3.2. Alarm selection 3.3. Prioritization 3.3.1. Prioritization procedure 3.3.2. Prioritization calculations 3.3.3. Internal prioritization review 3.4. Alarm set point determination 3.4.1. Alarm set point review 3.5. Alarm documentation 3.5.1. Alarm class 3.5.2. Highly managed alarms 3.6. Rationalization wrap-up and final approval 4. Detailed Design 4.1. Alarm attributes 4.2. Special alarm design considerations 4.3. Approved advanced alarm management techniques 4.4. HMI design guidance 5. Implementation, Operation & Maintenance 5.1. Implementation guidance 5.2. Operation 5.3. Alarm system maintenance 5.4. Training 5.5. Testing of alarms and alarm systems 6. Monitoring & Reporting 6.1. Alarm system metrics 6.2. Alarm system review 6.3. Alarm history preservation 7. Management of Change 8. Audit This organization of topics is general and applicable across industries. The detailed content, however, is specific to each pipeline operator or company. 2 In-house development versus outsourcing Developing an alarm management plan requires extensive input from internal sources regardless of whether the process is facilitated in-house or by an external consultant. Your organization s particular needs will determine if in-house development is a good option or if outsourcing will ultimately serve you better. If you envisage that your organization is going to develop a number of alarm management plans and expend a sustained effort on redesigning alarm systems, developing an in-house team of alarm management experts may be the best way to proceed. Although this does require an ongoing financial commitment in the form of training and salary, it may be cheaper in the long run than the cost of external consulting. However, you may still benefit from a limited engagement of an outside consultant, particularly if the effort requires skills that your staff does not currently possess. The consultant can bring the skills you need into your organization and work alongside your people to help them acquire the new skill set. In addition, a consultant can add a fresh perspective that can challenge internal preconceptions and an impartiality that can alleviate turf issues. 9 copyright J Bogdan Consulting LLC 2010 All Rights reserved

If your organization is responding on a more limited basis to regulatory requirements or is operating with limited personnel, hiring an external consultant will offer those same benefits and is likely to be the most timely and cost-effective way to accomplish your alarm management objectives. In-house efforts often strain available resources and, therefore, drag on unsuccessfully and expensively. An external consultant can, in approximately one week on-site, train a core group in the basics of alarm management and guide that group through a workshop to extract the information required for the alarm management plan. As a further benefit, the consultant s expertise and experience can help the team begin thinking about ways to use the alarm system as a rich source of operations and safety improvement ideas. The actual writing of your customized plan will take the consultant about another week off-site. In two weeks, you can have a completed plan. The time savings and improved results often make outsourcing the more cost-effective option. Whether the effort is in-house or aided by outsourcing, the ideal facilitator for the development of an alarm management plan will have experience in the following areas: Development of alarm management plans; Industrial experience in automation, engineering, and operations; (It is helpful, but not essential, for the facilitator to have experience in the organization s particular field.) Leadership and conflict resolution skills. 3 Thinking beyond the minimum Your motivation to write an alarm management plan may be to meet looming PHMSA regulations, but preparing a well-designed plan now will pay off in 2013 when the mandate to have it implemented takes effect. It involves almost no extra time or effort to think beyond the minimum requirements and prepare a plan that will be the foundation for continuing operational improvement. Broadening the scope of alarms to be reviewed is one area to consider. The PHMSA regulation specifies review of only safety-related alarms in Section e(1), but, in Section e(5), it requires a yearly evaluation of the general activity for each controller to assure there is sufficient time to analyze and react to incoming alarms. Whether or not PHMSA S intent was to have the plan cover only safety-related alarms, it is recommended that your plan specify review of all alarms for the following reasons: It is difficult to isolate safety-related alarms. Common methods of identifying them by alarm type or service class often result in misclassification of non-safety alarms as safety, and vice versa. Reviewing all alarms ensures that every safety-related alarm is identified. All alarms, not just safety-related, demand a response from the controller. Therefore, all alarms must be properly prioritized relative to each other for the controller to be able to respond appropriately to a series of alarms. Most importantly for your bottom line, reviewing all alarms enables you to eliminate unnecessary alarms. In a typical system, elimination of 30 to 60 percent of existing alarms is achievable. This not only improves safety by reducing controller workload, but also greatly simplifies the alarm system. 10 copyright J Bogdan Consulting LLC 2010 All Rights reserved

Another area in which thinking beyond PHMSA s minimum requirements is likely to be profitable is monitoring and assessment. This task often seems overwhelming because of the number of alarms. However, once you have eliminated unnecessary alarms, monitoring and assessment of the alarm system becomes practical for improving operations. To be of the most value, more frequent monitoring than required by PHMSA is necessary, but much of it can be done automatically. It is also essential that the task of assessing the collected data be clearly assigned and thoroughly carried out. Monitoring and assessment of frequent alarms may reveal an underlying, recurring problem that can be corrected. Consequential alarms can identify causal relationships between parts of the operation that are not readily apparent. Chattering alarms, which are typically considered nuisances to be eliminated, can indicate that conditions have shifted so that the operation is running undesirably close to an alarm limit. Every activated alarm is a valuable source of information, which, if tapped, can lead to significant operational gain. For example, one pipeline operator used the number of communication failure alarms to help decide the order in which to upgrade communication links from satellite to 3G. Another operator monitored the number of alarms generated by an old tank gauging system and compared them to those generated by a new one to determine that replacing the old system was desirable. Conclusion The mandate to develop an alarm management plan may seem at first glance to be a lowbenefit, high-nuisance proposition. However, alarms are not simply notifications to the controller that action is required; they are also notification to the rest of the organization that there is a potential operating or safety improvement to be made. A carefully crafted alarm management plan will provide the backbone for an alarm system that will yield rewards for years to come. 11 copyright J Bogdan Consulting LLC 2010 All Rights reserved