David Stymiest, PE, CHFM, CHSP, FASHE cell Copyright 2013, Smith Seckman Reid, Inc.; All rights reserved

Transcription

1 2013 Florida AHCA Seminar Expanding the Concept of Emergency Power Reliability David Stymiest, PE, CHFM, CHSP, FASHE cell Copyright 2013, Smith Seckman Reid, Inc.; All rights reserved NFPA Disclaimer Although the speaker is Chairman of the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed in this presentation are purely those of the speaker and shall not be considered the official position of NFPA or any of its Technical Committees and shall not be considered to be, nor be relied upon as, a Formal Interpretation. Readers are encouraged to refer to the entire texts of all referenced documents. NFPA members can obtain staff interpretations of NFPA standards at DStymiest@ssr inc.com 1

2 Main topics Reducing EP vulnerabilities Improving caregiver communication Planning for different types of failures Finding common mode failure potential Differences between EP reliability, availability & dependability, and what they mean to us Importance of EP equipment maintenance Other lessons learned ASHE survey: Utility outage duration ASHE super storm Sandy survey had 390 responses Number of Facilit ties (1 138 had utility out ages) (25 < 4 hrs) Hours of Electrical Utility Outage DStymiest@ssr inc.com 2

3 ASHE survey: EES unanticipated outage? YES YES 9% 9% NO 91% NO 91% YES = 24 NO = 246 ASHE survey: cause of EES failure 2 Causes of EES Failure Fuel System Other Cooling System Batteries Flooding/Rain Other: 7 Other includes 1 each: Breaker Failure, Electrical Fire, Oil Leak, Water Pump, Fan Bearing, Generator, and Required emergency items on NP DStymiest@ssr inc.com 3

4 Emergency Power Lessons RE learned Reliability Availability Dependability Lessons RE-learned Things break Ask critical questions Pay attention to the details Analyze impact of what if scenarios Importance of testing & maintenance Commonalities & history Common-mode failures Comprehensive vulnerability analyses inc.com 4

5 Other lessons RE-learned Basic emergency management concepts Ran out of fuel oil, no replacement fuel oil Lack of cellular communications could not reach service companies to request help Generator service could not reach facility Staff not trained to make portable generator connections FEMA realities during an emergency More lessons RE-learned Ongoing testing & maintenance are crucial Generation, Switching, Distribution Don t forget other utilities You can t control what you can t control So plan for it Without information you have only opinions The details will get you sweat the small stuff DStymiest@ssr inc.com 5

6 Sweat the small stuff Details are important Things break Communication is vital Not just about the weather for a long time 1999 State of Wash. gasoline pipeline rupture ~ 2009 DC Metro train crash P.E. 5/10 April 8, 2009 USA Today DStymiest@ssr inc.com 6

7 and what about this bit of old news? Managing Vulnerabilities Finding Pi Prioritizingiti i Assessing Reporting Mitigating Verifying DStymiest@ssr inc.com 7

8 Simplified hospital power system Common-mode failures Failures of two or more components or systems due to a single event or cause A safety engineering concept: once a failure mode is identified, it usually can be mitigated by adding extra or redundant equipment to the system The existence of an uncorrected common mode failure potentially removes the advantage of other redundancies. You cannot correct what you have not yet identified. DStymiest@ssr inc.com 8

9 Look beneath the surface Duplex equipment & common mode The two fuel oil pumps on this duplex pump set mitigate the impact of a single pump failure, but potential failures can occur due to common location or a single power circuit to the control panel. DStymiest@ssr inc.com 9

10 Common locations Paralleled generator sets can mitigate the impact of a single generator failure, but also can be subject to common mode failures due to shared location, shared fuel, or shared cooling systems. Causes of fuel oil contamination Natural fuel degradation from aging Day tank corrosion Clogged or fouled fuel oil filter Excessive fuel oil filter replacement interval Workmanship during fuel oil system renovation Fuel oil truck operator error Day tank microorganism contamination Inconsistent fuel oil quality from the supplier Incorrect biocide usage Inadequate sampling techniques. DStymiest@ssr inc.com 10

11 Paralleling Switchgear All generator outputs connected together Potential common-mode failure Control power failure Internal short circuit (no GF protection) Low probability but very high impact May become apparent when EPSS is energized next. Other types of common-mode failures Normal and emergency power equipment on same level Fuel oil storage tank subject to flooding Common fuel oil transfer pumps, controls, power circuits Feeders for elevated equipment located in flooded levels Other types of damage also inc.com 11

12 Service contractor observations Monthly testing work-arounds (VFD issues) came back to haunt facilities Single spin-on fuel filters clogged when Extended electrical utility blackout Dirty fuel was delivered Or clean fuel delivered to nearly empty tank stirred up bottom sediment Filters had extremely small micron level Put in multiple filter assemblies with isolating valves and bypass valves Other types of common-mode failures Co-located equipment and systems One sump pump Multiple sump pumps on same branch Transfer switch failure inc.com 12

13 Maintenance improves dependability Bypass isolation transfer switches can be maintained without turning off their loads, improving i operational dependability. Other common issues Communications with caregivers Some clinical personnel believe EP is or should be uninterruptible, should never fail. Misunderstandings: unrealistic expectations Medical journal article: usually less than 1- second duration upon loss of commercial power Different types of failures Different responses for each Updated failure procedures More pervasive, more complex systems now DStymiest@ssr inc.com 13

14 Communicating with caregivers Types of failures Normal down with emergency power working 1 emergency power branch down, normal working, other branches working 1 CB down with other CB still working Total electrical failure Simultaneously Cascading events speaker to request teaching slides DStymiest@ssr inc.com 14

15 Explaining Normal vs. Emergency Power Normal Outlet Generator is usually off Red (Emerg.) Outlet Emergency Generator Communicating NP switchboard outage What will happen? Power that will not be available (utility power fed through that switchboard) Selected normal lighting Selected NP receptacles (white & brown face) Selected equipment served from normal power Power that will be available (on generator) Power fed through other normal power switchboards Emergency lighting (includes egress lighting) Emergency receptacles (red face) Equipment served from emergency power inc.com 15

16 Explaining Normal vs. Emergency Power Simplified Emergency Power Supply System Generator on Red (Emerg.) Outlet Emergency Generator Communicating EP branch outage What will happen? Power that will be available (utility power) Normal lighting Normal receptacles (white & brown face) Equipment served from normal power Power that will not be available (load side of selected transfer switch[es]) Emergency lighting (includes egress lighting) Emergency receptacles (red face) Equipment served from emergency power inc.com 16

17 Contingency plans: stay cool under pressure What can go wrong? Then what happens? What response to use? Decisions Flexibility vs. details Planning for Internal Failures Must consider different failure points, not just at the mains. The responses will be different for each type of failure. It is TOO LATE to formulate a response after the failure has occurred. DStymiest@ssr inc.com 17

18 Lessons learned from 2005 disasters Reinforced lessons from Sept. 11, 2001 Common-mode failures Extended utility failures Extra backup lighting needed Operational flexibility needed Simultaneous failures of multiple utilities 911 and cellular systems disrupted Most common generator failures Starting system problems Fuel oil system problems Cooling system problems Installation error / lack of acceptance testing Inadequate maintenance Overloads generators, breakers, fuses Load shed malfunctions multiple generators fail DStymiest@ssr inc.com 18

19 Other causes of generator failures Lightning power surge damages generator controls Generator auxiliaries on normal power (fans, fuel transfer pumps) Failures during routine testing (thrown engine rod, fuel hose rupture) Other mechanical or electrical failures Generator breaker trips lack of protective coordination Planning for internal electrical failures Contingency planning: details important Doesn t need to be long Just needs to be correct for templates DStymiest@ssr inc.com 19

20 Simple contingency plan UTILITY FAILURE Normal Electrical Power Failure EMERGENCY CONDITIONS and BASIC STAFF RESPONSE BUILDING UTILITY FAILURES WHAT TO EXPECT Power only to emergency lights and RED plug outlets. WHAT TO DO Open Disaster Bin for flashlight, extension cords, batteries, etc. Know areas on emergency power. EMERGENCY CLINICAL INTERVENTIONS Ensure that Life Support Systems are attached to RED plugs; be prepared to handventilate. List clinical interventions Emergency Electrical Power Failure (only) Power only to normal lighting, and gray or white plug outlets Open Disaster Bin for flashlight, extension cords, batteries, etc. Check all patient care equipment and patient task lighting. Ensure that Life Support Systems are attached to gray/white plugs or to BACKUP red plugs if available; be prepared to hand ventilate. List clinical interventions Some examples - generators DStymiest@ssr inc.com 20

21 Some examples - switchboards Example of ATS failure procedure ATS 1 (EB) failure during business hours: Notify Administration, Security and Nursing Manager on duty Perform infection control risk assessments as required and take appropriate actions as required by the ICRA AHU2-1 failure: deploy fans or spot coolers to floor 2 Air compressor 1 & 2 failure: Use portable 120V air compressor to operate dampers as necessary Elevator failures: Security was already notified 2 nd floor isolation room exhaust fan: Nursing was already notified EQ subpanels (EQA, etc.) affected equipment AHU4 1: deploy fans and spot coolers to floor 4 Cafeteria walk in coolers and freezers: dietary to keep those doors closed Generator room lights Chiller bypass valve: can be manually operated DStymiest@ssr inc.com 21

22 Example of fuel oil pump failure procedure Supply pumps failure (power loss from CPEB 2 circuits from the same panel) Assess/troubleshoot the nature of the failure. Refer to electrical safety policy. Notify Director, who will notify others as necessary. Call electrical contractor for emergency service. ILSM/ALSM analysis is required (Generators not able to run beyond day tank fuel capacity constitute a Life Safety Code deficiency); ILSMs/ALSMs as determined, documentation Refer to Generator Failure Procedures because each generator can run only until its day tank empties If this failure disrupts fuel oil to the fire pump for more than 4 hours, notify FD & commence a documented fire watch. Complete utility failure incident report Facility Director reports incident to EOC Committee Revisit SEA-37 Vulnerability Analyses Preventing adverse events caused by emergency electrical power system failures published by TJC 9/6/2006 Also in TJC s 9/2007 EC News Recent events: Should we address the vulnerability analysis again perhaps more comprehensively this time? DStymiest@ssr inc.com 22

23 So what do we want from EP? Our emergency power systems need to power What they must When they must For as long as they must And we need to be able to roll with the punches when things go wrong A new paradigm Reliability Probability that system operates and gives the same result on successive trials Availability Probability that system will function at any instant required, including the next instant, and for as long as required from that point Dependability Measures availability, reliability & maintenance support DStymiest@ssr inc.com 23

24 Availability Consider this If no facilities system can guarantee 100% reliability, can any facilities system assure 100% percent availability? Common metric for large data centers 4 nines facility availability % How does your power system compare with data center power system design? Recommended approach to vulnerabilities 1. Consider each component that must operate; 2. Determine what scenarios will cause it to fail, including all What if? scenarios that could damage the power sources or feeders that t keep it running; 3. Compare those scenarios with others that may take out other redundant components, redundant power sources or redundant feeders; 4. Investigate all the possible causes of those scenarios, including commonalities in power sources, feeders or controls; 5. Address all resulting vulnerabilities that have been identified. DStymiest@ssr inc.com 24

25 Preparedness for power failures Things break Sweat the small stuff Small issues can take out systems Different failures When to plan? EP Vulnerability Analysis: Infrastructure For infrastructure, look at features, components, condition, locations, operating flexibility, spares, maintenance histories, vulnerabilities for Electrical service and NP distribution to ATS s EPSS and its auxiliary subsystems (FO +) ATS s, feeders, branches EP System documentation, labeling, failure procedures, test results, training DStymiest@ssr inc.com 25

26 EP Vulnerability Analysis: Power Sources NP system: major distribution EP system, Gen, ATS, ATS sources Sortinfrastructure systems, facility areas and facility services by each power train By each main switchboard, generator, ATS Determine where single equipment failures or wiring/feeder failures can take out redundant mechanical systems, areas or functions. Common mode failure vulnerability analysis. EP Vulnerability Analysis: Areas For all functional areas, look at: Higher vulnerability from infrastructure analysis (such as with less reliable equipment; poorer documentation, lack of power failure procedures, training, etc.) Higher vulnerability from common o mode failure vulnerability analysis All other vulnerability assessment tools DStymiest@ssr inc.com 26

27 EP system vulnerabilities: examples Common-mode failure potential ATS's not maintained regularly Because not bypass-isolation type ATS s not transferred every month Lack of branch maintenance Life Safety Branch Critical Branch Equipment System (Branch) Vulnerability analysis results Preparedness Activities Additional capacity Emergency equipment Identify additional resources More training / testing Contingency planning Mitigation Activities Policies & procedures Change in process Maintenance program (Start planning electrical shutdowns) Infrastructure t repair / upgrade Tighten rooms Leak detection DStymiest@ssr inc.com 27

28 Gap Analysis for VULNERABILITIES Gap Analysis can also address results of Vulnerability Analysis How vulnerable is EP System to failures? How vulnerable is NP System to failures? Where are the common-mode failure vulnerabilities? What can we do to mitigate these vulnerabilities: short term; long term? Gap analysis for SUPPLIED SERVICES Examples: fuel oil supplier, generator or ATS service company, spare parts supplier Understand any vulnerability or overcommitment with the service provider. Systematically identify gaps between where the supplier s crisis management capabilities end and your contingency plans begin. (EM ) Fix them. DStymiest@ssr inc.com 28

29 Proactive power system maintenance Predictive Maintenance (PdM) Condition-based Preventive Maintenance (PM) Calendar-based Reliability-Centered Maintenance (RCM) Based on system analyses, logic, statistical input, and criticality of equipment to be maintained Optimum mix of reactive, time-interval-based, condition-based, and proactive maintenance practices PdM Examples: Infrared Thermography Many facilities already scan NP equipment Include generator panel, paralleling li switchgear, and transfer switches in IR scanning scope of work. Make sure to scan equipment paralleling switchgear when it is energized. Thermal cycling works electrical lugs loose. Scan EPS (and EP lugs in ATS s) during monthly EPSS tests. inc.com 29

30 Designing for infrared thermography Pinhole lens technology can see through ½ viewport Some other PdM examples Diesel generator fluid testing Fuel oil Lubricating oil, Cooling water Rotating equipment vibration analysis Ultrasonic analysis inc.com 30

31 Electrical room maintenance Learn from weekly inspections Cleaning rooms: minimize contaminants finding their way inside the electrical equipment Change the filters Inspect for evidence of water Water and electricity are a mixture that no facility engineer wants Leak detection in electrical rooms Warns of water-based vulnerabilities when relocation is not practical Mech/Elect co-locations Elevation issues External water Internal piping Broken sumps DStymiest@ssr inc.com 31

32 Some risk assessment considerations Transfer switches maintenance history Normal power operational history Normal power maintenance shutdowns User and facility management action plans EP system maintenance shutdowns Thermographic scanning results Infrastructure conditions Competency training for maintainers Responses to various internal failures Responses to simultaneous multiple utility failures Operation of different equipment, not just the same equipment every month Understand and look for second order consequences inc.com 32

33 Shutdown lessons learned Where were extension cords required? What special precautions did Users take? What surprised you? What surprised the Units? Update contingency plans Utility Management Plan; P&P s EOP, Action Plans Input to next capital budget cycle Tracers on Preparing for Power Failures Test your own readiness Clinical equipment & personnel responses Reliance on UPS s Power shutdowns Maintenance EP loading Equipment failures Documentation ASHE paper with power failure tracers available upon request DStymiest@ssr inc.com 33

34 Thank you. Questions? David Stymiest, CHFM, CHSP, FASHE (Registered P.E. in LA, MS, MA) Cell inc.com 34

35 Managing Electrical Systems for Reliability David Stymiest, PE CHFM CHSP FASHE Senior Consultant Smith Seckman Reid, Inc. Nashville, TN This document was first presented at the 50 th ASHE annual conference, July 2013, Atlanta, GA. Copyright 2013 ASHE; contact ASHE at for reprint restrictions and permission.

36 Managing Electrical Systems for Reliability David Stymiest, PE CHFM CHSP FASHE inc.com, NFPA Disclaimer Although the speaker is Chairman of the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed in this paper are purely those of the speaker and shall not be considered the official position of NFPA or any of its Technical Committees and shall not be considered to be, nor be relied upon as, a Formal Interpretation. Refer to the entire texts of all referenced documents. Topics This white paper will focus on finding and assessing hidden vulnerabilities in power systems, the most typical types of hospital emergency power (EP) common mode failures, and reducing or eliminating their effects. We will also review the differences between power system reliability, availability and dependability. And finally we will assess the impact of different types of operation and maintenance (O&M) practices, testing, electrical safety issues, and contingency plans. Recent disasters Super storm Sandy and others During super storm Sandy, most hospital power emergency systems functioned as designed. But there were also some failures. Super storm Sandy caused power system failures such as some of those listed below. When previous power system failures occurred, they were also the result of one or more of the following, or even other causes not listed below: Normal wear and tear Required equipment not having backup power Natural events (weather, etc.) exceeding the protection of man made defenses Other types of events (terrorism, cyber hacking) exceeding the protection of man made defenses Lack of maintenance Common mode causes Insufficient training Communications difficulties Transportation difficulties Do sweat the small stuff We need to remember that things break. Sometimes those things will adversely affect other things. An example is a failed electrical connection that results in a large electrical fault (short circuit) that grows and destroys nearby equipment. This sort of thing is not supposed to occur but it does.

37 Sometimes it occurs because things have changed, sometimes because electrical maintenance is not being performed, and sometimes because personnel are not permitted to turn off electrical systems before working on them. Because many elements of our facilities interrelate it is necessary to sweat the small stuff. In other words, we need to pay attention to the small details. As an example, flooding can occur from both external and internal causes. External causes include rover flooding and train effect rain from a hurricane, but they can also include a contractor s backhoe breaking into an underground water main near the hospital. Internal causes can include a broken sprinkler head on an upper floor, a leaking domestic water line, or a broken chilled water pipe. Many hospitals have multiple fuel oil storage tanks, but those tanks may have common supply piping to the generators. It is common to find duplex fuel oil pump skids, but those duplex pump skids may be installed with single control panels and single control power circuits. A fuel oil pumping system might have its control power from one branch and the pump power from a second branch, perhaps requiring that both branches function correctly in order for the generator(s) to receive fuel oil. It is important to ask critical questions such as What will happen if this pipe breaks? or What might happen if the power circuit to this sump pump fails? This sort of thinking, followed by appropriate mitigating actions, is often necessary to prevent common mode conditions from causing common mode failures. Many failures are caused by normal wear and tear. The required weekly inspections and monthly testing of emergency power supply systems (EPSSs) can help facility personnel to find and fix incipient failures before the next normal power outage. From NFPA , Chapter 8 Routine Maintenance and Operational Testing: 8.4 Operational Inspection and Testing * EPSSs, including all appurtenant components, shall be inspected weekly and exercised under load at least monthly. Note that the above requirement for a weekly inspection extends beyond just the generator sets when you consider the scope of an EPSS: From NFPA , Chapter 3 Definitions: 3.3.4* Emergency Power Supply System (EPSS). A complete functioning EPS system coupled to a system of conductors, disconnecting means and overcurrent protective devices, transfer switches, and all control, supervisory, and support devices [emphasis added] up to and including the load terminals of the transfer equipment needed for the system to operate as a safe and reliable source of electric power. Hospitals are generally very good at inspecting the generator sets, but not all hospitals include all of the transfer switches, the panelboards between the generators and the transfer switches, the fuel oil system components, or even the remote alarm panels, in those weekly inspections. Since the remote alarm panel is relied upon to notify personnel of an EPSS derangement, shouldn t the weekly inspection include operating the audible alarm and the lamp test feature on that panel? Regular inspections of normal power systems can have a similar beneficial purpose. Regularly walking through electrical equipment rooms (both emergency and normal power equipment) and observing their conditions can alert facility personnel to pending problems before they adversely affect equipment operation.

38 Electrical equipment rooms that are subject to flooding should be considered for the application of leak detection equipment. The leak detection equipment, if installed, should send alarms to a critical alarm system of building automation system. That alarm feature should be regularly tested. Hospitals should consider adding the verification of that alarm feature into a regular inspection routine. The required EPSS weekly inspection for rooms that contain EPSS equipment could also include such items as verifying the operation of the leak detection equipment and alarms protecting those rooms. Power system dependability relies in part on the degree of maintenance. Maintenance is intended to overcome the impact of normal wear and tear on system components. We all know of the ongoing high level discussions between CMS and The Joint Commission regarding what constitutes the required maintenance of hospital critical utility systems and components. But there is sometimes more to the story. As an example, the maintenance practices for all EPSS equipment (remember the EPSS definition above) must be based upon: From NFPA , Chapter 8 Routine Maintenance and Operational Testing: The routine maintenance and operational testing program shall be based on all of the following [emphasis added]: (1) Manufacturer s recommendations (2) Instruction manuals (3) Minimum requirements of this chapter (4) The authority having jurisdiction The EPSS failures that occurred during and as a result of Hurricane Sandy were really not lessons learned, rather taking into account an informed historical perspective they were lessons relearned. They have all occurred before some on September 11, 2001; some during 2004 when five hurricanes hit the State of Florida; some during Hurricanes Katrina and Rita in 2005, and some during the numerous other natural disasters that have occurred throughout the United States during the past 20 years. Many of those failures were common mode failures. Common mode failures Common mode failures are failures of two or more structures, systems or components in the same manner or mode due to a single event or cause. The failure is considered to occur upon the loss of function whether that system or component is needed or not at that time. Unfortunately, a failure in a backup system (such as the EPSS) may not be apparent until the system is called upon to function, either during testing or on the failure of the system it is backing up (such as normal power.) A safety engineering concept considered by many organizations is that an identified failure mode can usually be mitigated by adding extra or redundant equipment to the system. However the existence of an uncorrected common mode failure potential might remove the advantage of such redundancies. Furthermore, you cannot correct what you have not yet identified. Considering the example of the duplex fuel oil pump set discussed above. Although this arrangement can mitigate the failure of a single pump, common mode failures have occurred due to common

39 location or a single power circuit to the control panel. Another example is with redundant equipment in common locations, such as with paralleled generator sets all in the same generator room. The paralleled generator arrangement can mitigate the impact of a single generator failure, but also can be subject to shared location, shared fuel or shared cooling system failures. Other types of potential common mode failure conditions include: Contaminated fuel oil system Normal and emergency power equipment located on same level Fuel oil storage tank subject to flooding Feeders or control circuits for elevated equipment located in levels subject to flooding or other single cause damage One sump pump protecting critical power equipment Multiple sump pumps protecting that critical power equipment, but all pumps are powered from the same panel or the same circuit Paralleling switchgear failure Transfer switch failure Our emergency power systems need to power what they must, when they must, and for as long as they must. And we need to be able to roll with the punches when things go wrong. This means having effective utility failure protocols. A new paradigm We have historically considered power system reliability somewhat simplistically. work. But consider the following safety/reliability engineering concepts: The system has to Reliability The probability that a system operates and gives the same result on successive trials Availability The probability that a system will function at any instant required, including the next instant, and for as long as required from that point Dependability This feature measures availability, reliability & maintenance support These issues are not at all simplistic. If no facilities system can guarantee 100% reliability, can any facilities system assure 100% percent availability? A common metric used when designing and operating large data centers reflects 4 nines (or 99.99%) facility availability. How does your power system compare with data center power system design? It is well recognized that maintenance improves dependability. As an example, bypass isolation transfer switches can be maintained without turning off their loads, improving operational dependability. But many hospitals do not have these features in their automatic transfer switches because they are more costly, larger than the simpler alternatives, and not required by codes and standards. Refer to the attached HFM Magazine article for a more comprehensive discussion of this issue.

40 For all hospital power systems, it is important to always follow all electrical safety precautions (refer to the latest edition of NFPA 70E for details) and also take steps to reduce the probability of damaging electrical short circuits (also called faults.) These steps include including regular testing and maintenance of the electrical power equipment. A review of the literature on the issue of electrical safety from short circuits and arc flashes indicates the following facts. Electrical faults at the 480 volt level and higher can involve very dangerous arc flash conditions. An arc flash is basically a short circuit through the air. It can be very destructive and dangerous. The arc plasma center can reach 35,000 F, which is 3 to 4 times the temperature on surface of sun. Arc flash can cause an event known as arc blast, which is when a large amount of concentrated heat and blast energy explodes outward from the equipment. This event can impact hearing because sound levels can exceed 160 db. Because gaseous copper is 67,000 times its solid volume, molten metal can be expelled from the electrical equipment at greater than 700 MPH. An arc flash can also result in extreme pressures (thousands of pounds per square feet) on anyone close to the occurrence. Finding and Mitigating Vulnerabilities It often takes a comprehensive vulnerability analysis coupled with risk assessments to identify all of the potential common mode failures. Vulnerabilities should be determined at the following, including location based issues: Electrical utility lines and service equipment Main normal power switchgear, major normal power distribution switchboards, and major normal power risers Emergency generators and all subsystems (fuel oil, cooling, starting, etc.) and appurtenances Emergency power ATSs, distribution feeders, panels, etc. Essential electrical system branches Once we find vulnerabilities, we want to consider options for mitigating them. Such options could include relocating equipment, although that could be costly, time consuming, and even impractical. Other mitigating efforts could include installing leak detection, increasing or improving maintenance, and following up on operating anomalies. Common communications issues Facilities personnel should be communicating the potential types of failures that can occur with the hospital s caregivers. Some clinical personnel believe emergency power is or should be uninterruptible, that it should never fail. However nice that situation would be, it is an unrealistic expectation. Electrical systems (both normal and emergency) are more pervasive now, and more complex, than they were 10 or 20 years ago when many existing utility failure procedures were written. There are different types of failure that can occur, and those failures often require different responses. The response is different for each of the following scenarios: Normal power goes down while emergency power is still available. This is the most commonly mentioned electrical failure (and often the ONLY mentioned electrical failure) in many hospital electrical failure procedures. This failure mode can be the result of an offsite utility failure but can also be the result of an internal failure such as an electrical short circuit. One Essential Electrical System (EES) branch fails although normal power is still available and the

41 other EES branches are still available. This can occur as a result of core drilling within the hospital, an ATS failure, a panelboard failure, or even a motor or emergency lighting ballast ground fault (short circuit to ground) that is not isolated low enough within that portion of the power system. For operating rooms or other areas that are fortunate enough to be served by two separate critical branches, one critical branch could be down (again due to an internal failure) while the remaining critical branch is still providing power. A total electrical failure can occur, sometimes simultaneously such as in the case of a tornado that damages the normal and emergency power equipment rooms, but more often as cascading failures such as those that occurred during Hurricane Sandy and many other natural disasters. References After the Storm Expanding the concept of emergency power reliability by David Stymiest. Original content published in the January 2013 issue of HFM magazine, Vol. 26, No. 1, 2013 by Health Forum Inc. All rights reserved. Permission granted to SSR for digital use only. NFPA Edition Addresses Generator Fuel Oil Management in SSR Compliance News, Sep Oct 2012 edition, cfm articles.blogspot.com/2012/10/compliance news nfpa edition.html ASHE White Paper: Planning for Power Failures originally presented at the 2007 ASHE Annual Conference. inc.com/pdfs/planning%20for%20power%20failures_david%20stymiest_ashe%20pape r.pdf Electrical Power Failure in the Operating Room: A Neglected Topic in Anesthesia Safety, John H. Eichhorn, M.D., and Eugene A. Hessel II, M.D., Anesthesia & Analgesia, vol. 110, no. 6 June 2010) : analgesia.org/content/110/6/1519 Preventing adverse events caused by emergency electrical power system failures, The Joint Commission Sentinel Event Alert, Issue 37, Sept. 6, 2006: rgency_electrical_power_system_failures/ Sounding a Sentinel Event Alert on Emergency Electrical Power Systems Environment of Care News September 2007: Averting Common Causes of Generator Failure (Part 2), Darren Dembski and Sarah Escalante, Facilities Engineering Journal, November/December 2009: Generator Fan Failure Triggered AWS Outage, Rich Miller, Data Center Knowledge blog June 21, 2012: outage/

42 Multiple Generator Failures Caused Amazon Outage, Rich Miller, Data Center Knowledge blog July 3, 2012: failures caused amazonoutag e/ Super Bowl XLVII blackout: Power redundancy, paralleling and LED lighting, Steve Taranovich EDN Network, March 28, 2013, management/ /super Bowl XLVII blackout Power redunda ncy paralleling and LED lighting Response to a Partial Power Failure in the Operating Room, Tammy Carpenter, M.D., and Stephen T. Robinson, M.D., Anesthesia & Analgesia, vol. 110, no. 6 (June 2010) : analg.com/content/110/6/1644.full.pdf+html Averting Common Causes of Generator Failure (Part 1), Darren Dembski and Sarah Escalante, Facilities Engineering Journal, September/ October 2009: Problems Encountered During Hurricane Sandy by Dan Chisholm, MGI Systems, Inc., 3/4/13. encountered during hurricane sandy/ Managing Hospital Emergency Power Systems Testing, Operation, Maintenance and Power Failure Planning, ASHE management monograph, 2006 (update planned for fall/winter 2013): NFPA Standard for Emergency and Standby Power Systems, NFPA; NFPA 70E 2012 Handbook Handbook for Electrical Safety in the Workplace, 3 rd Edition, Edited by Jeffrey S. Sargent and Michael D. Fontaine,