A PRESENTATION FOR TAHFM 2013 Interlink NFPA 110 Update + Expanding the Concept of EP Reliability David Stymiest, PE, CHFM, CHSP, FASHE (PE in LA, MS, MA) DStymiest@ssr-inc.com, cell 504.232.1113 Copyright 2013, Smith Seckman Reid, Inc., Nashville, TN
NFPA Disclaimer Although the speaker is Chairman of the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed in this presentation are purely those of the speaker and shall not be considered the official position of NFPA or any of its Technical Committees and shall not be considered to be, nor be relied upon as, a Formal Interpretation. Readers are encouraged to refer to the entire texts of all referenced documents. NFPA members can obtain staff interpretations of NFPA standards at www.nfpa.org.
Learning Objectives Discuss NFPA 110-2013 current status List lessons learned from recent natural disasters List what else we can take away? Describe how we can make EP systems more dependable
Major 110 Changes in 2013 Fuel oil ASTM, maintenance, storage, tanks, quality, sampling, testing Relaxed 96 hour seismic language Testing NFPA 70B Reference
NFPA 110 major changes 1999 to 2013
Major 111 NFPA Changes for 2013 Exclude Level 1 SEPSS from high energy normal power room This does NOT apply to non-sepss UPSs Add DC Rectifier Plants to get ready for upcoming changes in telecom industry that will bring more DC systems into facilities
Want to make a change? Both standards are in the Annual 2015 update cycle. Public input closing date: July 8, 2013 www.nfpa.org/110 www.nfpa.org/111
Go to any NFPA code page For example, for 110: www.nfpa.org/110
www.nfpa.org/110 Next edition tab
NFPA 110 & NFPA 111 Revision Cycle The next edition of this standard is now open for Public Input (formerly proposals). Revision cycle information Revision Cycle: Annual 2015 Revised Edition Date: 2016 First Draft (previously Report on Proposals (ROP)) Public Input Closing Date: 7/8/2013 Public Input form (word) or Submit Public Input online First Draft Report Posting Date: 3/7/2014 Second Draft (previously Report on Comments (ROC)) Public Comment Closing Date: 5/16/2014 Second Draft Report Posting Date: 1/16/2015 Notice of Intent to Make a Motion (NITMAM) NITMAM Closing Date: 3/6/2015 NITMAM Posting Date: 5/1/2015
NFPA 110 & 111 Public Input
EP Reliability Original content published in the January 2013 issue of HFM magazine, Vol. 26, No. 1, 2013 by Health Forum Inc. All rights reserved. Reprint permission granted to SSR for digital use only
Lessons learned Pay attention to details Analyze impact of what if scenarios Things break Critical questions to ask Importance of testing & maintenance Commonalities between emergencies Vulnerability analyses Common-mode failures
Other Lessons Learned Basic EM Concepts Ran out of fuel oil, no replacement oil Lack of cellular communications could not reach service companies to request help EP service company could not reach facility Staff not trained to make portable generator connections
General Issues Communications with caregivers Some clinical personnel believe EP is or should be uninterruptible, should never fail. Misunderstandings: unrealistic expectations Medical journal article: usually less than 1- second duration upon loss of commercial power Different types of failures Different responses Updated failure procedures More pervasive, more complex systems now
Types of Failures Normal down with emergency power online 1 emergency power branch down, normal online 1 CB down with other CB still online Total electrical failure Simultaneously Cascading events
Explaining Normal vs. Emergency Power Normal Outlet Red (Emerg.) Outlet Generator is usually off Emergency Generator
Explaining Normal vs. Emergency Power Simplified Emergency Power Supply System Generator on Red (Emerg.) Outlet Emergency Generator
Hospital Emergency Power Supply System (Before NFPA 99-2012 brought back EB) (Before NFPA 99-2012)
What will happen? Power that will not be available (utility power fed through that switchboard) Selected normal lighting Selected NP receptacles (white & brown face) Selected equipment served from normal power Power that will be available (on generator) Power fed through other normal power switchboards Emergency lighting (includes egress lighting) Emergency receptacles (red face) Communicate the Impact: Outage of a NP switchboard Equipment served from emergency power
Communicate the Impact: Emergency power branch outage What will happen? Power that will be available (utility power) Normal lighting Normal receptacles (white & brown face) Equipment served from normal power Power that will not be available (load side of selected transfer switch[es]) Emergency lighting (includes egress lighting) Emergency receptacles (red face) Equipment served from emergency power
Life safety branch outage Emergency lighting Exit signs Fire alarms Medical gas and vacuum alarms Emergency communication systems Generator (& ATS?) room lighting/outlets Elevator cab lighting, commun., signals Automatic doors in means of egress Generator fuel pump?
Critical branch outage EP (red) outlets Selected task lighting in many pt care areas Critical care areas: PACUs, ICUs, ORs, ERs, procedure rooms, etc. Medication prep / dispensing areas Nurse call system Telephone equipment rooms/closets Generator fuel pump? Central med/surg suction systems/controls?
Equipment branch outage (NFPA 99-2012 nomenclature) Critical care area ventilation Isolation room exhaust fans Hazardous exhaust fans Central med/surg suction systems/controls? Sump pumps Compressed air systems Smoke control, stair pressurization Kitchen hood supply/exhaust Heating, Jockey pump, elevators,
So what do we want from EP? Our emergency power systems need to power What they must When they must For as long as they must And we need to be able to roll with the punches when things go wrong
A new paradigm? Reliability Probability that system operates and gives the same result on successive trials Availability Probability that system will function at any instant required, including the next instant, and for as long as required from that point Dependability Measures availability, reliability & maintenance support
Availability Consider this If no facilities system can guarantee 100% reliability, can any facilities system assure 100% percent availability? Common metric for large data centers 4 nines facility availability - 99.99% How does your power system compare with data center power system design?
Consider revisiting TJC s Sentinel Event Alert 37 Preventing adverse events caused by emergency electrical power system failures published by TJC 9/6/2006 Also in TJC s 9/2007 EC News Recent events: Should we address the vulnerability analysis again perhaps more comprehensively this time?
Common-mode failures Failures of two or more components or systems due to a single event or cause A safety engineering concept: once a failure mode is identified, it usually can be mitigated by adding extra or redundant equipment to the system The existence of an uncorrected common mode failure potentially removes the advantage of other redundancies. You cannot correct what you have not yet identified.
Duplex equipment can have common mode failures The two fuel oil pumps on this duplex pump set mitigate the impact of a single pump failure, but potential failures can occur due to common location or a single power circuit to the control panel.
Redundant equipment in common locations Paralleled generator sets can mitigate the impact of a single generator failure, but also can be subject to common mode failures due to shared location, shared fuel or cooling systems.
Major Changes in NFPA 110 1999 to 2010 Category / Topic Protection from hazards 1999 Edition (Ref. by 2000 LSC) Rooms, shelters, or separate buildings located to minimize the possibility of damage from flooding (fire fighting, sewer, similar) 2010 Edition (Ref. by 2012 LSC) Rooms, shelters, or separate buildings designed and located to minimize the damage from flooding (fire fighting, sewer, similar) 2013 Edition Rooms, enclosures, or separate buildings designed and located to minimize the damage from flooding (fire fighting, sewer, other)
Other types of common-mode failures Contaminated fuel oil system Normal and emergency power equipment on same level Fuel oil storage tank subject to flooding Common fuel oil transfer pumps, controls, power circuits Feeders for elevated equipment located in flooded levels Other types of damage also
Other types of common-mode failures One sump pump Multiple sump pumps on same branch Transfer switch failure Paralleling switchgear failure
Maintenance improves dependability Bypass isolation transfer switches can be maintained without turning off their loads, improving operational dependability.
Major Changes in NFPA 110 1999 to 2010 Category / Topic Stipulated generator maintenance Stipulated ATS maintenance 1999 Edition (Ref. by 2000 LSC) None stated. All EPSS per manufacturer. Suggest Annex if no mfr. Basic All EPSS per manufacturer. 2010 Edition (Ref. by 2012 LSC) SAME No change; Major annual PM in Annex 2013 Edition SAME, but also references NFPA 70B for info SAME, but also references NFPA 70B for info
Major Changes in NFPA 110 1999 to 2010 Category / Topic Stipulated paralleling gear maintenance Stipulated battery maintenance 1999 Edition (Ref. by 2000 LSC) None stated. All EPSS per manufacturer. Weekly inspections; follow manufacturer s specs 2010 Edition (Ref. by 2012 LSC) Similar to ATS stipulated maintenance Also permits battery conductance testing in lieu of specific gravity 2013 Edition SAME, but also references NFPA 70B for info SAME, but also references NFPA 70B for info
Lessons Learned Ongoing testing & maintenance are crucial Generation, Switching, Distribution Don t forget other utilities You can t control what you can t control So plan for it Without information you have only opinions The details will get you sweat the small stuff
Recommended Approach 1. Consider each component that must operate; 2. Determine what scenarios will cause it to fail, including all What if? scenarios that could damage the power sources or feeders that keep it running; 3. Compare those scenarios with others that may take out other redundant components, redundant power sources or redundant feeders; 4. Investigate all the possible causes of those scenarios, including commonalities in power sources, feeders or controls; 5. Address all of the resulting vulnerabilities that have been identified.
Contingency plans for failures Brainstorm for all potential failures. What can go wrong? Have a contingency plan for each. Equipment failures Generator Transfer switch; panel; circuit breaker Include (older) breaker failures Update call lists for emergency suppliers
Simplest contingency plan very basic info can be too simplistic for many failures EMERGENCY CONDITIONS and BASIC STAFF RESPONSE UTILITY FAILURE BUILDING UTILITY FAILURES WHAT TO EXPECT WHAT TO DO OTHER RESPONSES Normal Electrical Power Failure Power only to emergency lights and RED plug outlets. Open Disaster Bin for flashlight, extension cords, batteries, etc. Know areas on emergency power. Ensure that Life Support Systems are attached to RED plugs; be prepared to handventilate. Report to Supervisor.
Simplest contingency plan very basic info can be too simplistic for many failures EMERGENCY CONDITIONS and BASIC STAFF RESPONSE BUILDING UTILITY FAILURES UTILITY FAILURE WHAT TO EXPECT WHAT TO DO EMERGENCY CLINICAL INTERVENTIONS Normal Electrical Power Failure Power only to emergency lights and RED plug outlets. Open Disaster Bin for flashlight, extension cords, batteries, etc. Know areas on emergency power. Ensure that Life Support Systems are attached to RED plugs; be prepared to handventilate. List clinical interventions Emergency Electrical Power Failure (only) Power only to normal lighting, and gray or white plug outlets Open Disaster Bin for flashlight, extension cords, batteries, etc. Check all patient care equipment and patient task lighting. Ensure that Life Support Systems are attached to gray/white plugs or to BACKUP red plugs if available; be prepared to hand-ventilate. List clinical interventions
Preparedness for power failures Recognize that failures will occur Plan appropriate responses ahead of time Response will be different for each failure It is too late to formulate a response after the failure occurs
Detailed emergency procedures help us to keep cool under pressure
A helpful resource from ASHE Shutdown preparation Similar to planning for internal electrical outages
Critical Operations Power Systems Article 708, starting in 2008 NEC Article 708 probably does NOT apply to you; consider it as an example of best practices. Risk assessment: ID hazards, mitigation strategies Documented load testing & maintenance Commissioning Surge protection, selective coordination, bypassisolation ATS s, selective load pickup & shedding, 2 levels GF if req d. 72 hours onsite fuel, means to connect portable genset Physical security, physical protection, separation, 1 hr & 2 hr FRR, location limitations, 100 year floodplain considered, labeling, fire protection etc. NEC and National Electrical Code are registered trademarks of the National Fire Protection Association,
Thank you. Questions anyone? David Stymiest, P.E. CHFM CHSP FASHE (P.E. in LA, MS, MA) Cell 504.232.1113 DStymiest@ssr-inc.com (References follow this slide.)
References 1 After the Storm - Expanding the concept of emergency power reliability by David Stymiest. Original content published in the January 2013 issue of HFM magazine, Vol. 26, No. 1, 2013 by Health Forum Inc. All rights reserved. Permission granted to SSR for digital use only. http://tinyurl.com/hfmafterthestorm ASHE White Paper: NFPA 110 / 111 Changes for 2013 originally presented at the 2012 ASHE Annual Conference. https://www.softconference.com/ashe/sessiondetail.asp?sid=280670 NFPA 110-2013 Edition Addresses Generator Fuel Oil Management in SSR Compliance News, Sep-Oct 2012 edition, http://ssr-cfmarticles.blogspot.com/2012/10/compliance-news-nfpa-110-2013-edition.html ASHE White Paper: Planning for Power Failures originally presented at the 2007 ASHE Annual Conference. http://www.ssrinc.com/pdfs/planning%20for%20power%20failures_david%20stymiest_ashe %20Paper.pdf Response to a Partial Power Failure in the Operating Room, Tammy Carpenter, M.D., and Stephen T. Robinson, M.D., Anesthesia & Analgesia, vol. 110, no. 6 (June 2010) 1644 46: www.anesth-analg.com/content/110/6/1644.full.pdf+html
References 2 Electrical Power Failure in the Operating Room: A Neglected Topic in Anesthesia Safety, John H. Eichhorn, M.D., and Eugene A. Hessel II, M.D., Anesthesia & Analgesia, vol. 110, no. 6 June 2010) 1519 21: www.anesthesiaanalgesia.org/content/110/6/1519 Preventing adverse events caused by emergency electrical power system failures, The Joint Commission Sentinel Event Alert, Issue 37, Sept. 6, 2006: www.jointcommission.org/sentinel_event_alert_issue_37_preventing_adverse_ events_caused_by_emergency_electrical_power_system_failures/ Sounding a Sentinel Event Alert on Emergency Electrical Power Systems Environment of Care News September 2007: www.jcrinc.com Averting Common Causes of Generator Failure (Part 1), Darren Dembski and Sarah Escalante, Facilities Engineering Journal, September/ October 2009: www.afe.org/publications/genfailure09.09.pdf Problems Encountered During Hurricane Sandy by Dan Chisholm, MGI Systems, Inc., 3/4/13. http://mgisys.com/problems-encountered-during-hurricane-sandy/
References 3 Averting Common Causes of Generator Failure (Part 2), Darren Dembski and Sarah Escalante, Facilities Engineering Journal, November/December 2009: www.afe.org/publications/journal/generatorfailure2.pdf Generator Fan Failure Triggered AWS Outage, Rich Miller, Data Center Knowledge blog June 21, 2012: www.datacenterknowledge.com/archives/2012/06/21/aws-outage/ Multiple Generator Failures Caused Amazon Outage, Rich Miller, Data Center Knowledge blog July 3, 2012: www.datacenterknowledge.com/archives/2012/07/03/multiplegenerator-failures-causedamazonoutage/ Managing Hospital Emergency Power Systems Testing, Operation, Maintenance and Power Failure Planning, ASHE management monograph, 2006: www.ashe.org/resources/management_monographs/mg2009stymiest.html NFPA 110-1999, NFPA 110-2010, and NFPA 110-2013 Standard for Emergency and Standby Power Systems, NFPA; www.nfpa.org/110