5. How and Why the Blackout Began in Ohio

Transcription

1 5. How and Why the Blackout Began in Ohio Summary This chapter explains the major events electrical, computer, and human that occurred as the blackout evolved on August 14, 2003, and identifies the causes of the initiation of the blackout. The period covered in this chapter begins at 12:15 Eastern Daylight Time (EDT) on August 14, 2003 when inaccurate input data rendered MISO s state estimator (a system monitoring tool) ineffective. At 13:31 EDT, FE s Eastlake 5 generation unit tripped and shut down automatically. Shortly after 14:14 EDT, the alarm and logging system in FE s control room failed and was not restored until after the blackout. After 15:05 EDT, some of FE s 345-kV transmission lines began tripping out because the lines were contacting overgrown trees within the lines right-of-way areas. By around 15:46 EDT when FE, MISO and neighboring utilities had begun to realize that the FE system was in jeopardy, the only way that the blackout might have been averted would have been to drop at least 1,500 MW of load around Cleveland and Akron. No such effort was made, however, and by 15:46 EDT it may already have been too late for a large load-shed to make any difference. After 15:46 EDT, the loss of some of FE s key 345-kV lines in northern Ohio caused its underlying network of 138-kV lines to begin to fail, leading in turn to the loss of FE s Sammis-Star 345-kV line at 16:06 EDT. The chapter concludes with the loss of FE s Sammis-Star line, the event that triggered the uncontrollable 345 kv cascade portion of the blackout sequence. The loss of the Sammis-Star line triggered the cascade because it shut down the 345-kV path into northern Ohio from eastern Ohio. Although the area around Akron, Ohio was already blacked out due to earlier events, most of northern Ohio remained interconnected and electricity demand was high. This meant that the loss of the heavily overloaded Sammis-Star line instantly created major and unsustainable burdens on lines in adjacent areas, and the cascade spread rapidly as lines and generating units automatically tripped by protective relay action to avoid physical damage. Chapter Organization This chapter is divided into several phases that correlate to major changes within the FirstEnergy system and the surrounding area in the hours leading up to the cascade: Phase 1: A normal afternoon degrades Phase 2: FE s computer failures Phase 3: Three FE 345-kV transmission line failures and many phone calls Phase 4: The collapse of the FE 138-kV system and the loss of the Sammis-Star line. Key events within each phase are summarized in Figure 5.1, a timeline of major events in the origin of the blackout in Ohio. The discussion that follows highlights and explains these significant events within each phase and explains how the events were related to one another and to the cascade. Specific causes of the blackout and associated recommendations are identified by icons. Phase 1: A Normal Afternoon Degrades: 12:15 EDT to 14:14 EDT Overview of This Phase Northern Ohio was experiencing an ordinary August afternoon, with loads moderately high to serve air conditioning demand, consuming high levels of reactive power. With two of Cleveland s active and reactive power production anchors already shut down (Davis-Besse and Eastlake 4), the loss of the Eastlake 5 unit at 13:31 EDT further depleted critical voltage support for the Cleveland-Akron area. Detailed simulation modeling reveals that the loss of Eastlake 5 was a significant factor in the outage later that afternoon with Eastlake 5 out of service, transmission line U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 45

2 Figure 5.1. Timeline: Start of the Blackout in Ohio loadings were notably higher but well within normal ratings. After the loss of FE s Harding-Chamberlin line at 15:05 EDT, the system eventually became unable to sustain additional contingencies, even though key 345 kv line loadings did not exceed their normal ratings. Had Eastlake 5 remained in service, subsequent line loadings would have been lower. Loss of Eastlake 5, however, did not initiate the blackout. Rather, subsequent computer failures leading to the loss of situational awareness in FE s control room and the loss of key FE transmission lines due to contacts with trees were the most important causes. At 14:02 EDT, Dayton Power & Light s (DPL) Stuart-Atlanta 345-kV line tripped off-line due to a tree contact. This line had no direct electrical effect on FE s system but it did affect MISO s performance as reliability coordinator, even though PJM is the reliability coordinator for the DPL line. One of MISO s primary system condition evaluation tools, its state estimator, was unable to assess system conditions for most of the period between 12:15 and 15:34 EDT, due to a combination of human error and the effect of the loss of DPL s Stuart-Atlanta line on other MISO lines as reflected in the state estimator s calculations. Without an effective state estimator, MISO was unable to perform contingency analyses of generation and line losses within its reliability zone. Therefore, through 15:34 EDT MISO could not determine that with Eastlake 5 down, other transmission lines would overload if FE lost a major transmission line, and could not issue appropriate warnings and operational instructions. In the investigation interviews, all utilities, control area operators, and reliability coordinators indicated that the morning of August 14 was a reasonably typical day. 1 FE managers referred to it as peak load conditions on a less than peak load day. Dispatchers consistently said that while voltages were low, they were consistent with historical voltages. 2 Throughout the morning and early afternoon of August 14, FE reported a growing need for voltage support in the upper Midwest. 46 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

3 The FE reliability operator was concerned about low voltage conditions on the FE system as early as 13:13 EDT. He asked for voltage support (i.e., increased reactive power output) from FE s interconnected generators. Plants were operating in automatic voltage control mode (reacting to system voltage conditions and needs rather than constant reactive power output). As directed in FE s Manual of Operations, 3 the FE reliability operator began to call plant operators to ask for additional voltage support from their units. He noted to most of them that system voltages were sagging all over. Several mentioned that they were already at or near their reactive output limits. None were asked to reduce their real power output to be able to produce more reactive output. He called the Sammis plant at 13:13 EDT, West Lorain at 13:15 EDT, Eastlake at 13:16 EDT, made three calls to unidentified plants between 13:20 EDT and 13:23 EDT, a Unit 9 at 13:24 EDT, and two more at 13:26 EDT and 13:28 EDT. 4 The operators worked to get shunt capacitors at Avon that were out of service restored to support voltage, 5 but those capacitors could not be restored to service. Following the loss of Eastlake 5 at 13:31 EDT, FE s operators concern about voltage levels increased. They called Bay Shore at 13:41 EDT and Perry at Energy Management System (EMS) and Decision Support Tools Operators look at potential problems that could arise on their systems by using contingency analyses, driven from state estimation, that are fed by data collected by the SCADA system. SCADA: System operators use System Control and Data Acquisition systems to acquire power system data and control power system equipment. SCADA systems have three types of elements: field remote terminal units (RTUs), communication to and between the RTUs, and one or more Master Stations. Field RTUs, installed at generation plants and substations, are combination data gathering and device control units. They gather and provide information of interest to system operators, such as the status of a breaker (switch), the voltage on a line or the amount of real and reactive power being produced by a generator, and execute control operations such as opening or closing a breaker. Telecommunications facilities, such as telephone lines or microwave radio channels, are provided for the field RTUs so they can communicate with one or more SCADA Master Stations or, less commonly, with each other. Master stations are the pieces of the SCADA system that initiate a cycle of data gathering from the field RTUs over the communications facilities, with time cycles ranging from every few seconds to as long as several minutes. In many power systems, Master Stations are fully integrated into the control room, serving as the direct interface to the Energy Management System (EMS), receiving incoming data from the field RTUs and relaying control operations commands to the field devices for execution. State Estimation: Transmission system operators must have visibility (condition information) over their own transmission facilities, and recognize the impact on their own systems of events and facilities in neighboring systems. To accomplish this, system state estimators use the real-time data measurements available on a subset of those facilities in a complex mathematical model of the power system that reflects the configuration of the network (which facilities are in service and which are not) and real-time system condition data to estimate voltage at each bus, and to estimate real and reactive power flow quantities on each line or through each transformer. Reliability coordinators and control areas that have them commonly run a state estimator on regular intervals or only as the need arises (i.e., upon demand). Not all control areas use state estimators. Contingency Analysis: Given the state estimator s representation of current system conditions, a system operator or planner uses contingency analysis to analyze the impact of specific outages (lines, generators, or other equipment) or higher load, flow, or generation levels on the security of the system. The contingency analysis should identify problems such as line overloads or voltage violations that will occur if a new event (contingency) happens on the system. Some transmission operators and control areas have and use state estimators to produce base cases from which to analyze next contingencies ( N-1, meaning normal system minus 1 key element) from the current conditions. This tool is typically used to assess the reliability of system operation. Many control areas do not use real time contingency analysis tools, but others run them on demand following potentially significant system events. U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 47

4 Figure 5.2. Timeline Phase 1 13:43 EDT to ask the plants for more voltage support. Again, while there was substantial effort to support voltages in the Ohio area, FirstEnergy personnel characterized the conditions as not being unusual for a peak load day, although this was not an all-time (or record) peak load day. 6 Key Phase 1 Events 1A) 12:15 EDT to 16:04 EDT: MISO s state estimator software solution was compromised, and MISO s single contingency reliability assessment became unavailable. 1B) 13:31:34 EDT: Eastlake Unit 5 generation tripped in northern Ohio. 1C) 14:02 EDT: Stuart-Atlanta 345-kV transmission line tripped in southern Ohio. 1A) MISO s State Estimator Was Turned Off: 12:15 EDT to 16:04 EDT It is common for reliability coordinators and control areas to use a state estimator (SE) to improve the accuracy of the raw sampled data they have for the electric system by mathematically processing raw data to make it consistent with the electrical system model. The resulting information on equipment voltages and loadings is used in software tools such as real time contingency analysis (RTCA) to simulate various conditions and outages to evaluate the reliability of the power system. The RTCA tool is used to alert operators if the system is operating insecurely; it can be run either on a regular schedule (e.g., every 5 minutes), when triggered by some system event (e.g., the loss of a power plant or transmission line), or when initiated by an operator. MISO usually runs the SE every 5 minutes, and the RTCA less frequently. If the model does not have accurate and timely information about key pieces of system equipment or if key input data are wrong, the state estimator may be unable to reach a solution or it will reach a solution that is labeled as having a high degree of error. In August, MISO considered its SE and RTCA tools to be still under development and not fully mature; those systems have since been completed and placed into full operation. On August 14 at about 12:15 EDT, MISO s state estimator produced a solution with a high mismatch (outside the bounds of acceptable error). This was traced to an outage of Cinergy s Bloomington-Denois Creek 230-kV line although it was out of service, its status was not updated in MISO s state estimator. Line status information within MISO s reliability coordination area is transmitted to MISO by the ECAR data network or direct links and is intended to be automatically linked to the SE. This requires coordinated data naming as well as instructions that link the data to the tools. For this line, the automatic linkage of line status to the state estimator had not yet been established. The line status was corrected and MISO s analyst obtained a good SE solution at 13:00 EDT and an RTCA solution at 13:07 EDT. However, to troubleshoot this problem the analyst had turned off the automatic trigger that runs the state estimator every five minutes. After fixing the problem he forgot to re-enable it, so although he had successfully run the SE and RTCA manually to reach a set of correct system analyses, the tools were not returned to normal automatic operation. Thinking the system had been successfully restored, the analyst went to lunch. 48 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

5 Cause 4 RC Diagnostic Support The fact that the state estimator was not running automatically on its regular 5-minute schedule was discovered about 14:40 EDT. The automatic trigger was re-enabled but again the state estimator failed to solve successfully. This time investigation identified the Stuart-Atlanta 345-kV line outage (which occurred at 14:02 EDT) to be the likely cause. This line is within the Dayton Power and Light control area in southern Ohio and is under PJM s reliability umbrella rather than MISO s. Even though it affects electrical flows within MISO, its status had not been automatically linked to MISO s state estimator. The discrepancy between actual measured system flows (with Stuart-Atlanta off-line) and the MISO model (which assumed Stuart-Atlanta on-line) prevented the state estimator from solving correctly. At 15:09 EDT, when informed by the system engineer that the Stuart-Atlanta line appeared to be the problem, the MISO operator said (mistakenly) that this line was in service. The system engineer then tried unsuccessfully to reach a solution with the Stuart-Atlanta line modeled as in service until approximately 15:29 EDT, when the MISO operator called PJM to verify the correct status. After they determined that Stuart-Atlanta had tripped, they updated the state estimator and it solved successfully. The RTCA was then run manually and solved successfully at 15:41 EDT. MISO s state estimator and contingency analysis were back under full automatic operation and solving effectively by 16:04 EDT, about two minutes before the start of the cascade. In summary, the MISO state estimator and real time contingency analysis tools were effectively out of service between 12:15 EDT and 16:04 EDT. This prevented MISO from promptly performing precontingency early warning assessments of power system reliability over the afternoon of August 14. Recommendations 3, page 143; 6, page 147; 30, page 163 1B) Eastlake Unit 5 Tripped: 13:31 EDT Eastlake Unit 5 (rated at 597 MW) is in northern Ohio along the southern shore of Lake Erie, connected to FE s 345-kV transmission system (Figure 5.3). The Cleveland and Akron loads are generally supported by generation from a combination of the Eastlake, Perry and Davis-Besse units, along with significant imports, particularly from 9,100 MW of generation located along the Ohio and Pennsylvania border. The unavailability of Eastlake 4 and Davis-Besse meant that FE had to import more energy into the Cleveland-Akron area to support its load. When Eastlake 5 dropped off-line, replacement power transfers and the associated reactive power to support the imports to the local area contributed to the additional line loadings in the region. At 15:00 EDT on August 14, FE s load was approximately 12,080 MW, and they were importing about 2,575 MW, 21% of their total. FE s system reactive power needs rose further. Cause 1 System Understanding The investigation team s system simulations indicate that the loss of Eastlake 5 was a critical step in the sequence of events. Contingency analysis simulation of the conditions following the loss of the Harding-Chamberlin 345-kV circuit at 15:05 EDT showed that the system would be unable to sustain some contingencies without line overloads above emergency ratings. However, when Eastlake 5 was modeled as in service and fully available in those simulations, all overloads above emergency limits were eliminated, even with the loss of Harding- Chamberlin. Cause 2 Situational Awareness FE did not perform a contingency analysis after the loss of Eastlake 5 at 13:31 EDT to determine whether the loss of further lines or plants would put their system at risk. FE also did not perform a contingency analysis after the loss of Harding-Chamberlin at 15:05 EDT (in part because they did not know that it had tripped out of service), nor does the utility routinely conduct such studies. 7 Thus FE did not discover that their system was no longer in an N-1 Figure 5.3. Eastlake Unit 5 Recommendation 23, page 160 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 49

6 secure state at 15:05 EDT, and that operator action was needed to remedy the situation. 1C) Stuart-Atlanta 345-kV Line Tripped: 14:02 EDT Cause 1 System Understanding Recommendations 3, page 143, 22, page 159 The Stuart-Atlanta 345-kV transmission line is in the control area of Dayton Power and Light. At 14:02 EDT the line tripped due to contact with a tree, causing a short circuit to ground, and locked out. Investigation team modeling reveals that the loss of DPL s Stuart-Atlanta line had no significant electrical effect on power flows and voltages in the FE area. The team examined the security of FE s system, testing power flows and voltage levels with the combination of plant and line outages that evolved on the afternoon of August 14. This analysis shows that the availability or unavailability of the Stuart-Atlanta 345-kV line did not change the capability or performance of FE s system or affect any line loadings within the FE system, either immediately after its trip or later that afternoon. The only reason why Stuart-Atlanta matters to the blackout is because it contributed to the failure of MISO s state estimator to operate effectively, so MISO could not fully identify FE s precarious system conditions until 16:04 EDT. 8 Data Exchanged for Operational Reliability The topology of the electric system is essentially the road map of the grid. It is determined by how each generating unit and substation is connected to all other facilities in the system and at what voltage levels, the size of the individual transmission wires, the electrical characteristics of each of those connections, and where and when series and shunt reactive devices are in service. All of these elements affect the system s impedance the physics of how and where power will flow across the system. Topology and impedance are modeled in power-flow programs, state estimators, and contingency analysis software used to evaluate and manage the system. Topology processors are used as front-end processors for state estimators and operational display and alarm systems. They convert the digital telemetry of breaker and switch status to be used by state estimators, and for displays showing lines being opened or closed or reactive devices in or out of service. A variety of up-to-date information on the elements of the system must be collected and exchanged for modeled topology to be accurate in real time. If data on the condition of system elements are incorrect, a state estimator will not successfully solve or converge because the real-world line flows and voltages being reported will disagree with the modeled solution. Data Needed: A variety of operational data is collected and exchanged between control areas and reliability coordinators to monitor system performance, conduct reliability analyses, manage congestion, and perform energy accounting. The data exchanged range from real-time system data, which is exchanged every 2 to 4 seconds, to OASIS reservations and electronic tags that identify individual energy transactions between parties. Much of these data are collected through operators SCADA systems. ICCP: Real-time operational data is exchanged and shared as rapidly as it is collected. The data is passed between the control centers using an Inter-Control Center Communications Protocol (ICCP), often over private frame relay networks. NERC operates one such network, known as NERCNet. ICCP data are used for minute-tominute operations to monitor system conditions and control the system, and include items such as line flows, voltages, generation levels, dynamic interchange schedules, area control error (ACE), and system frequency, as well as in state estimators and contingency analysis tools. IDC: Since the actual power flows along the path of least resistance in accordance with the laws of physics, the NERC Interchange Distribution Calculator (IDC) is used to determine where it will actually flow. The IDC is a computer software package that calculates the impacts of existing or proposed power transfers on the transmission components of the Eastern Interconnection. The IDC uses a power flow model of the interconnection, representing over 40,000 substation buses, 55,000 lines and transformers, and more than 6,000 generators. This model calculates transfer distribution factors (TDFs), which tell how a power transfer would load up each system (continued on page 51) 50 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

7 Phase 2: FE s Computer Failures: 14:14 EDT to 15:59 EDT Overview of This Phase Starting around 14:14 EDT, FE s control room operators lost the alarm function that provided audible and visual indications when a significant piece of equipment changed from an acceptable to a problematic condition. Shortly thereafter, the EMS system lost a number of its remote control consoles. Next it lost the primary server computer that was hosting the alarm function, and then the backup server such that all functions that were being supported on these servers were stopped at 14:54 EDT. However, for over an hour no one in FE s control room grasped that their computer systems were not operating properly, even though FE s Information Technology support staff knew of the problems and were working to solve them, and the absence of alarms and other symptoms offered many clues to the operators of the EMS system s impaired state. Thus, without a functioning EMS or the knowledge that it had failed, FE s system operators remained unaware that their electrical system condition was beginning to Data Exchanged for Operational Reliability (Continued) element, and outage transfer distribution factors (OTDFs), which tell how much power would be transferred to a system element if another specific system element were lost. The IDC model is updated through the NERC System Data Exchange (SDX) system to reflect line outages, load levels, and generation outages. Power transfer information is input to the IDC through the NERC electronic tagging (E-Tag) system. SDX: The IDC depends on element status information, exchanged over the NERC System Data Exchange (SDX) system, to keep the system topology current in its powerflow model of the Eastern Interconnection. The SDX distributes generation and transmission outage information to all operators, as well as demand and operating reserve projections for the next 48 hours. These data are used to update the IDC model, which is used to calculate the impact of power transfers across the system on individual transmission system elements. There is no current requirement for how quickly asset owners must report changes in element status (such as a line outage) to the SDX some entities update it with facility status only once a day, while others submit new information immediately after an event occurs. NERC is now developing a requirement for regular information update submittals that is scheduled to take effect in the summer of SDX data are used by some control centers to keep their topology up-to-date for areas of the interconnection that are not observable through direct telemetry or ICCP data. A number of transmission providers also use these data to update their transmission models for short-term determination of available transmission capability (ATC). E-Tags: All inter-control area power transfers are electronically tagged (E-Tag) with critical information for use in reliability coordination and congestion management systems, particularly the IDC in the Eastern Interconnection. The Western Interconnection also exchanges tagging information for reliability coordination and use in its unscheduled flow mitigation system. An E-Tag includes information about the size of the transfer, when it starts and stops, where it starts and ends, and the transmission service providers along its entire contract path, the priorities of the transmission service being used, and other pertinent details of the transaction. More than 100,000 E-Tags are exchanged every month, representing about 100,000 GWh of transactions. The information in the E-Tags is used to facilitate curtailments as needed for congestion management. Voice Communications: Voice communication between control area operators and reliability is an essential part of exchanging operational data. When telemetry or electronic communications fail, some essential data values have to be manually entered into SCADA systems, state estimators, energy scheduling and accounting software, and contingency analysis systems. Direct voice contact between operators enables them to replace key data with readings from the other systems telemetry, or surmise what an appropriate value for manual replacement should be. Also, when operators see spurious readings or suspicious flows, direct discussions with neighboring control centers can help avert problems like those experienced on August 14, U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 51

8 Figure 5.4. Timeline Phase 2 degrade. Unknowingly, they used the outdated system condition information they did have to discount information from others about growing system problems. Key Events in This Phase 2A) 14:14 EDT: FE alarm and logging software failed. Neither FE s control room operators nor FE s IT EMS support personnel were aware of the alarm failure. 2B) 14:20 EDT: Several FE remote EMS consoles failed. FE s Information Technology (IT) engineer was computer auto-paged. 2C) 14:27:16 EDT: Star-South Canton 345-kV transmission line tripped and successfully reclosed. 2D) 14:32 EDT: AEP called FE control room about AEP indication of Star-South Canton 345-kV line trip and reclosure. FE had no alarm or log of this line trip. 2E) 14:41 EDT: The primary FE control system server hosting the alarm function failed. Its applications and functions were passed over to a backup computer. FE s IT engineer was auto-paged. 2F) 14:54 EDT: The FE back-up computer failed and all functions that were running on it stopped. FE s IT engineer was auto-paged. Failure of FE s Alarm System Cause 2 Situational Awareness FE s computer SCADA alarm and logging software failed sometime shortly after 14:14 EDT (the last time that a valid alarm came in), after voltages had begun deteriorating but well before any of FE s lines began to contact trees and trip out. After that time, the FE control room consoles did not receive any further alarms, nor were there any alarms being printed or posted on the EMS s alarm logging facilities. Power system operators rely heavily on audible and on-screen alarms, plus alarm logs, to reveal any significant changes in their system s conditions. After 14:14 EDT on August 14, FE s operators were working under a significant handicap without these tools. However, they were in further jeopardy because they did not know that they were operating without alarms, so that they did not realize that system conditions were changing. Alarms are a critical function of an EMS, and EMS-generated alarms are the fundamental means by which system operators identify events on the power system that need their attention. Without alarms, events indicating one or more significant system changes can occur but remain undetected by the operator. If an EMS s alarms are absent, but operators are aware of the situation and the remainder of the EMS s functions are intact, the operators can potentially continue to use the EMS to monitor and exercise control of their power system. In such circumstances, the operators would have to do so via repetitive, continuous manual scanning of numerous data and status points located within the multitude of individual displays available within their EMS. Further, it would be difficult for the operator to identify quickly the most relevant of the many screens available. In the same way that an alarm system can inform operators about the failure of key grid facilities, it 52 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

9 can also be set up to alarm them if the alarm system itself fails to perform properly. FE s EMS did not have such a notification system. Although the alarm processing function of FE s EMS failed, the remainder of that system generally continued to collect valid real-time status information and measurements about FE s power system, and continued to have supervisory control over the FE system. The EMS also continued to send its normal and expected collection of information on to other monitoring points and authorities, including MISO and AEP. Thus these entities continued to receive accurate information about the status and condition of FE s power system after the time when FE s EMS alarms failed. FE s operators were unaware that in this situation they needed to manually and more closely monitor and interpret the SCADA information they were Alarms System operators must keep a close and constant watch on the multitude of things occurring simultaneously on their power system. These include the system s load, the generation and supply resources to meet that load, available reserves, and measurements of critical power system states, such as the voltage levels on the lines. Because it is not humanly possible to watch and understand all these events and conditions simultaneously, Energy Management Systems use alarms to bring relevant information to operators attention. The alarms draw on the information collected by the SCADA real-time monitoring system. Alarms are designed to quickly and appropriately attract the power system operators attention to events or developments of interest on the system. They do so using combinations of audible and visual signals, such as sounds at operators control desks and symbol or color changes or animations on system monitors, displays, or map boards. EMS alarms for power systems are similar to the indicator lights or warning bell tones that a modern automobile uses to signal its driver, like the door open bell, an image of a headlight high beam, a parking brake on indicator, and the visual and audible alert when a gas tank is almost empty. Power systems, like cars, use status alarms and limit alarms. A status alarm indicates the state of a monitored device. In power systems these are commonly used to indicate whether such items as switches or breakers are open or receiving. Continuing on in the belief that their system was satisfactory, lacking any alarms from their EMS to the contrary, and without visualization aids such as a dynamic map board or a projection of system topology, FE control room operators were subsequently surprised when they began receiving telephone calls from other locations and information sources MISO, AEP, PJM, and FE field operations staff who offered information on the status of FE s transmission facilities that conflicted with FE s system operators understanding of the situation. Recommendations 3, page 143, 22, page 159 Analysis of the alarm problem performed by FE suggests that the alarm process essentially stalled while processing an alarm event, such that the process began to run in a manner that failed to complete the processing of that alarm or closed (off or on) when they should be otherwise, or whether they have changed condition since the last scan. These alarms should provide clear indication and notification to system operators of whether a given device is doing what they think it is, or what they want it to do for instance, whether a given power line is connected to the system and moving power at a particular moment. EMS limit alarms are designed to provide an indication to system operators when something important that is measured on a power system device such as the voltage on a line or the amount of power flowing across it is below or above pre-specified limits for using that device safely and efficiently. When a limit alarm activates, it provides an important early warning to the power system operator that elements of the system may need some adjustment to prevent damage to the system or to customer loads like the low fuel or high engine temperature warnings in a car. When FE s alarm system failed on August 14, its operators were running a complex power system without adequate indicators of when key elements of that system were reaching and passing the limits of safe operation and without awareness that they were running the system without these alarms and should no longer assume that not getting alarms meant that system conditions were still safe and unchanging. U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 53

10 produce any other valid output (alarms). In the meantime, new inputs system condition data that needed to be reviewed for possible alarms built up in and then overflowed the process input buffers. 9,10 Loss of Remote EMS Terminals. Between 14:20 EDT and 14:25 EDT, some of FE s remote EMS terminals in substations ceased operation. FE has advised the investigation team that it believes this occurred because the data feeding into those terminals started queuing and overloading the terminals buffers. FE s system operators did not learn about this failure until 14:36 EDT, when a technician at one of the sites noticed the terminal was not working after he came in on the 15:00 shift, and called the main control room to report the problem. As remote terminals failed, each triggered an automatic page to FE s Information Technology (IT) staff. 11 The investigation team has not determined why some terminals failed whereas others did not. Transcripts indicate that data links to the remote sites were down as well. 12 EMS Server Failures. FE s EMS system includes several server nodes that perform the higher functions of the EMS. Although any one of them can host all of the functions, FE s normal system configuration is to have a number of host subsets of the applications, with one server remaining in a hot-standby mode as a backup to the others should any fail. At 14:41 EDT, the primary server hosting the EMS alarm processing application failed, due either to the stalling of the alarm application, queuing to the remote EMS terminals, or some combination of the two. Following preprogrammed instructions, the alarm system application and all other EMS software running on the first server automatically transferred ( failedover ) onto the back-up server. However, because the alarm application moved intact onto the backup while still stalled and ineffective, the backup server failed 13 minutes later, at 14:54 EDT. Accordingly, all of the EMS applications on these two servers stopped running. Cause 2 Situational Awareness Recommendation 22, page 159 The concurrent loss of both EMS servers apparently caused several new problems for FE s EMS and the operators who used it. Tests run during FE s after-the-fact analysis of the alarm failure event indicate that a concurrent absence of these servers can significantly slow down the rate at which the EMS system puts new or refreshes existing displays on operators computer consoles. Thus at times on August 14th, operators screen refresh rates the rate at which new information and displays are painted onto the computer screen, normally 1 to 3 seconds slowed to as long as 59 seconds per screen. Since FE operators have numerous information screen options, and one or more screens are commonly nested as sub-screens to one or more top level screens, operators ability to view, understand and operate their system through the EMS would have slowed to a frustrating crawl. 13 This situation may have occurred between 14:54 EDT and 15:08 EDT when both servers failed, and again between 15:46 EDT and 15:59 EDT while FE s IT personnel attempted to reboot both servers to remedy the alarm problem. Loss of the first server caused an auto-page to be issued to alert FE s EMS IT support personnel to the problem. When the back-up server failed, it too sent an auto-page to FE s IT staff. They did not notify control room operators of the problem. At 15:08 EDT, IT staffers completed a warm reboot (restart) of the primary server. Startup diagnostics monitored during that reboot verified that the computer and all expected processes were running; accordingly, FE s IT staff believed that they had successfully restarted the node and all the processes it was hosting. However, although the server and its applications were again running, the alarm system remained frozen and non-functional, even on the restarted computer. The IT staff did not confirm that the alarm system was again working properly with the control room operators. Another casualty of the loss of both servers was the Automatic Generation Control (AGC) function hosted on those computers. Loss of AGC meant that FE s operators could not run affiliated power plants on pre-set programs to respond automatically to meet FE s system load and interchange obligations. Although the AGC did not work from 14:54 EDT to 15:08 EDT and 15:46 EDT to 15:59 EDT (periods when both servers were down), this loss of function does not appear to have had an effect on the blackout. Recommendation 19, page 156 Recommendation 22, page 159 Cause 2 The concurrent loss of the EMS servers also caused the failure of Situational FE s strip chart function. There Awareness are many strip charts in the FE Reliability Operator control room driven by the EMS computers, showing a variety 54 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

11 of system conditions, including raw ACE (Area Control Error), FE system load, and Sammis-South Canton and South Canton-Star loading. These charts are visible in the reliability operator control room. The chart printers continued to scroll but because the underlying computer system was locked up the chart pens showed only the last valid measurement recorded, without any variation from that measurement as time progressed (i.e., the charts flat-lined ). There is no indication that any operators noticed or reported the failed operation of the charts. 14 The few charts fed by direct analog telemetry, rather than the EMS system, showed primarily frequency data, and remained available throughout the afternoon of August 14. These yield little useful system information for operational purposes. FE s Area Control Error (ACE), the primary control signal used to adjust generators and imports to match load obligations, did not function between 14:54 EDT and 15:08 EDT and later between 15:46 EDT and 15:59 EDT, when the two servers were down. This meant that generators were not controlled during these periods to meet FE s load and interchange obligations (except from 15:00 EDT to 15:09 EDT when control was switched to a backup controller). There were no apparent negative consequences from this failure. It has not been established how loss of the primary generation control signal was identified or if any discussions occurred with respect to the computer system s operational status. 15 EMS System History. The EMS in service at FE s Ohio control center is a GE Harris (now GE Network Systems) XA21 system. It was initially brought into service in Other than the application of minor software fixes or patches typically encountered in the ongoing maintenance and support of such a system, the last major updates or revisions to this EMS were implemented in On August 14 the system was not running the most current release of the XA21 software. FE had Who Saw What? What data and tools did others have to monitor the conditions on the FE system? Midwest ISO (MISO), reliability coordinator for FE Alarms: MISO received indications of breaker trips in FE that registered in MISO s alarms; however, the alarms were missed. These alarms require a look-up to link the flagged breaker with the associated line or equipment and unless this line was specifically monitored, require another look-up to link the line to the monitored flowgate. MISO operators did not have the capability to click on the on-screen alarm indicator to display the underlying information. Real Time Contingency Analysis (RTCA): The contingency analysis showed several hundred violations around 15:00 EDT. This included some FE violations, which MISO (FE s reliability coordinator) operators discussed with PJM (AEP s Reliability Coordinator). a Simulations developed for this investigation show that violations for a contingency would have occurred after the Harding-Chamberlin trip at 15:05 EDT. There is no indication that MISO addressed this issue. It is not known whether MISO identified the developing Sammis-Star problem. Flowgate Monitoring Tool: While an inaccuracy has been identified with regard to this tool it still functioned with reasonable accuracy and prompted MISO to call FE to discuss the Hanna-Juniper line problem. It would not have identified problems south of Star since that was not part of the flowgate and thus not modeled in MISO s flowgate monitor. AEP Contingency Analysis: According to interviews, b AEP had contingency analysis that covered lines into Star. The AEP operator identified a problem for Star-South Canton overloads for a Sammis- Star line loss about 15:33 EDT and asked PJM to develop TLRs for this. However, due to the size of the requested TLR, this was not implemented before the line tripped out of service. Alarms: Since a number of lines cross between AEP s and FE s systems, they had the ability at their respective end of each line to identify contingencies that would affect both. AEP initially noticed FE line problems with the first and subsequent trips of the Star-South Canton 345-kV line, and called FE three times between 14:35 EDT and 15:45 EDT to determine whether FE knew the cause of the outage. c a MISO Site Visit, Benbow interview. b AEP Site Visit, Ulrich interview. c Example at 14:35, Channel 4; 15:19, Channel 4; 15:45, Channel 14 (FE transcripts). U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 55

12 decided well before August 14 to replace it with one from another vendor. FE personnel told the investigation team that the alarm processing application had failed on occasions prior to August 14, leading to loss of the alarming of system conditions and events for FE s operators. 16 However, FE said that the mode and behavior of this particular failure event were both first time occurrences and ones which, at the time, FE s IT personnel neither recognized nor knew how to correct. FE staff told investigators that it was only during a post-outage support call with GE late on 14 August that FE and GE determined that the only available course of action to correct the alarm problem was a cold reboot 17 of FE s overall XA21 system. In interviews immediately after the blackout, FE IT personnel indicated that they discussed a cold reboot of the XA21 system with control room operators after they were told of the alarm problem at 15:42 EDT, but decided not to take such action because operators considered power system conditions precarious, were concerned about the length of time that the reboot might take to complete, and understood that a cold reboot would leave them with even less EMS functionality until it was completed. 18 Clues to the EMS Problems. There is an entry in FE s western desk operator s log at 14:14 EDT referring to the loss of alarms, but it is not clear whether that entry was made at that time or subsequently, referring back to the last known alarm. There is no indication that the operator mentioned the problem to other control room staff and supervisors or to FE s IT staff. Recommendation 33, page 164 Recommendation 26, page 161 The first clear hint to FE control room staff of any computer problems occurred at 14:19 EDT when a caller and an FE control room operator discussed the fact that three sub-transmission center dial-ups had failed. 19 At 14:25 EDT, a control room operator talked with a caller about the failure of these three remote EMS consoles. 20 The next hint came at 14:32 EDT, when FE scheduling staff spoke about having made schedule changes to update the EMS pages, but that the totals did not update. 21 Cause 2 Although FE s IT staff would have been aware that concurrent loss Situational of its servers would mean the loss Awareness of alarm processing on the EMS, the investigation team has found no indication that the IT staff informed the control room staff either when they began work on the servers at 14:54 EDT, or when they completed the primary server restart at 15:08 EDT. At 15:42 EDT, the IT staff were first told of the alarm problem by a control room operator; FE has stated to investigators that their IT staff had been unaware before then that the alarm processing sub-system of the EMS was not working. Without the EMS systems, the only remaining ways to monitor system conditions would have been through telephone calls and direct analog telemetry. FE control room personnel did not realize that alarm processing on their EMS was not working and, subsequently, did not monitor other available telemetry. Cause 2 Situational Awareness During the afternoon of August 14, FE operators talked to their field personnel, MISO, PJM (concerning an adjoining system in PJM s reliability coordination region), adjoining systems (such as AEP), and customers. The FE operators received pertinent information from all these sources, but did not recognize the emerging problems from the clues offered. This pertinent information included calls such as that from FE s eastern control center asking about possible line trips, FE Perry nuclear plant calls regarding what looked like nearby line trips, AEP calling about their end of the Star-South Canton line tripping, and MISO and PJM calling about possible line overloads. Recommendations 19, page 156; 26, page 161 Without a functioning alarm system, the FE control area operators failed to detect the tripping of electrical facilities essential to maintain the security of their control area. Unaware of the loss of alarms and a limited EMS, they made no alternate arrangements to monitor the system. When AEP identified the 14:27 EDT circuit trip and reclosure of the Star 345 kv line circuit breakers at AEP s South Canton substation, the FE operator dismissed the information as either not accurate or not relevant to his system, without following up on the discrepancy between the AEP event and the information from his own tools. There was no subsequent verification of conditions with the MISO reliability coordinator. Only after AEP notified FE that a 345-kV circuit had tripped and locked out did the FE control area operator compare this information to actual breaker conditions. FE failed to inform its reliability coordinator and adjacent control areas when they became aware that system conditions 56 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

13 had changed due to unscheduled equipment outages that might affect other control areas. Phase 3: Three FE 345-kV Transmission Line Failures and Many Phone Calls: 15:05 EDT to 15:57 EDT Overview of This Phase From 15:05:41 EDT to 15:41:35 EDT, three 345-kV lines failed with power flows at or below each transmission line s emergency rating. These line trips were not random. Rather, each was the result of a contact between a line and a tree that had grown so tall that, over a period of years, it encroached into the required clearance height for the line. As each line failed, its outage increased the loading on the remaining lines (Figure 5.5). As each of the transmission lines failed, and power flows shifted to other transmission paths, voltages on the rest of FE s system degraded further (Figure 5.6). Key Phase 3 Events Recommendations 26, page 161; 30, page 163 3A) 15:05:41 EDT: Harding-Chamberlin 345-kV line tripped. 3B) 15:31-33 EDT: MISO called PJM to determine if PJM had seen the Stuart-Atlanta 345-kV line outage. PJM confirmed Stuart-Atlanta was out. 3C) 15:32:03 EDT: Hanna-Juniper 345-kV line tripped. 3D) 15:35 EDT: AEP asked PJM to begin work on a 350-MW TLR to relieve overloading on the Star-South Canton line, not knowing the Hanna-Juniper 345-kV line had already tripped at 15:32 EDT. 3E) 15:36 EDT: MISO called FE regarding post-contingency overload on Star-Juniper 345-kV line for the contingency loss of the Hanna-Juniper 345-kV line, unaware at the start of the call that Hanna-Juniper had already tripped. 3F) 15:41:33-41 EDT: Star-South Canton 345-kV tripped, reclosed, tripped again at 15:41:35 EDT and remained out of service, all while AEP and PJM were discussing TLR relief options (event 3D). Transmission lines are designed with the expectation that they will sag lower when they become hotter. The transmission line gets hotter with heavier line loading and under higher ambient temperatures, so towers and conductors are designed to be tall enough and conductors pulled tightly enough to accommodate expected sagging and still meet safety requirements. On a summer day, conductor temperatures can rise from 60 C on mornings with average wind to 100 C with hot air temperatures and low wind conditions. A short-circuit occurred on the Harding- Chamberlin 345-kV line due to a contact between the line conductor and a tree. This line failed with power flow at only 44% of its normal and emergency line rating. Incremental line current and temperature increases, escalated by the loss of Harding-Chamberlin, caused more sag on the Hanna-Juniper line, which contacted a tree and failed with power flow at 88% of its normal and emergency line rating. Star-South Canton Figure 5.5. FirstEnergy 345-kV Line Flows Figure 5.6. Voltages on FirstEnergy s 345-kV Lines: Impacts of Line Trips U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 57

14 Figure 5.7. Timeline Phase 3 contacted a tree three times between 14:27:15 EDT and 15:41:33 EDT, opening and reclosing each time before finally locking out while loaded at 93% of its emergency rating at 15:41:35 EDT. Each of these three lines tripped not because of excessive sag due to overloading or high conductor temperature, but because it hit an overgrown, untrimmed tree. 22 Cause 3 Tree Trimming Overgrown trees, as opposed to excessive conductor sag, caused each of these faults. While sag may have contributed to these events, these incidents occurred because the trees grew too tall and encroached into the space below the line which is intended to be clear of any objects, not because the lines sagged into short trees. Because the trees were so tall (as discussed below), each of these lines faulted under system conditions well within specified operating parameters. The investigation team found field evidence of tree contact at all three locations, including human observation of the Hanna-Juniper contact. Evidence outlined below confirms that contact with trees caused the short circuits to ground that caused each line to trip out on August 14. To be sure that the evidence of tree/line contacts and tree remains found at each site was linked to the events of August 14, the team looked at whether these lines had any prior history of outages in preceding months or years that might have resulted in the burn marks, debarking, and other vegetative evidence of line contacts. The record establishes that there were no prior sustained outages known to be caused by trees for these lines in 2001, 2002, and Like most transmission owners, FE patrols its lines regularly, flying over each transmission line twice a year to check on the condition of the rights-of-way. Notes from fly-overs in 2001 and 2002 indicate that the examiners saw a significant number of trees and brush that needed clearing or Line Ratings A conductor s normal rating reflects how heavily the line can be loaded under routine operation and keep its internal temperature below a certain temperature (such as 90 C). A conductor s emergency rating is often set to allow higher-than-normal power flows, but to limit its internal temperature to a maximum temperature (such as 100 C) for no longer than a specified period, so that it does not sag too low or cause excessive damage to the conductor. For three of the four 345-kV lines that failed, FE set the normal and emergency ratings at the same level. Many of FE s lines are limited by the maximum temperature capability of its terminal equipment, rather than by the maximum safe temperature for its conductors. In calculating summer emergency ampacity ratings for many of its lines, FE assumed 90 F (32 C) ambient air temperatures and 6.3 ft/sec (1.9 m/sec) wind speed, a which is a relatively high wind speed assumption for favorable wind cooling. Actual temperature on August 14 was 87 F (31 C) but wind speed at certain locations in the Akron area was somewhere between 0 and 2 ft/sec (0.6 m/sec) after 15:00 EDT that afternoon. a FirstEnergy Transmission Planning Criteria (Revision 8), page U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

15 trimming along many FE transmission lines. Notes from fly-overs in the spring of 2003 found fewer problems, suggesting that fly-overs do not allow effective identification of the distance between a tree and the line above it, and need to be supplemented with ground patrols. Recommendations 16, page 154; 27, page 162 3A) FE s Harding-Chamberlin 345-kV Line Tripped: 15:05 EDT At 15:05:41 EDT, FE s Harding-Chamberlin line (Figure 5.8) tripped and locked out while loaded at 44% of its normal and emergency rating. At this low loading, the line temperature would not exceed safe levels even if still air meant there Utility Vegetation Management: When Trees and Lines Contact Vegetation management is critical to any utility company that maintains overhead energized lines. It is important and relevant to the August 14 events because electric power outages occur when trees, or portions of trees, grow up or fall into overhead electric power lines. While not all outages can be prevented (due to storms, heavy winds, etc.), some outages can be mitigated or prevented by managing the vegetation before it becomes a problem. When a tree contacts a power line it causes a short circuit, which is read by the line s relays as a ground fault. Direct physical contact is not necessary for a short circuit to occur. An electric arc can occur between a part of a tree and a nearby high-voltage conductor if a sufficient distance separating them is not maintained. Arcing distances vary based on such factors such as voltage and ambient wind and temperature conditions. Arcs can cause fires as well as short circuits and line outages. Most utilities have right-of-way and easement agreements allowing them to clear and maintain vegetation as needed along their lines to provide safe and reliable electric power. Transmission easements generally give the utility a great deal of control over the landscape, with extensive rights to do whatever work is required to maintain the lines with adequate clearance through the control of vegetation. The three principal means of managing vegetation along a transmission right-of-way are pruning the limbs adjacent to the line clearance zone, removing vegetation completely by mowing or cutting, and using herbicides to retard or kill further growth. It is common to see more tree and brush removal using mechanical and chemical tools and relatively less pruning along transmission rights-of-way. FE s easement agreements establish extensive rights regarding what can be pruned or removed a Standard language in FE s right-of-way easement agreement. b Utility Vegetation Management Final Report, CN Utility Consulting, March in these transmission rights-of-way, including: the right to erect, inspect, operate, replace, relocate, repair, patrol and permanently maintain upon, over, under and along the above described right of way across said premises all necessary structures, wires, cables and other usual fixtures and appurtenances used for or in connection with the transmission and distribution of electric current, including telephone and telegraph, and the right to trim, cut, remove or control by any other means at any and all times such trees, limbs and underbrush within or adjacent to said right of way as may interfere with or endanger said structures, wires or appurtenances, or their operations. a FE uses a 5-year cycle for transmission line vegetation maintenance (i.e., it completes all required vegetation work within a 5-year period for all circuits). A 5-year cycle is consistent with industry practices, and it is common for transmission providers not to fully exercise their easement rights on transmission rights-of-way due to landowner or land manager opposition. A detailed study prepared for this investigation, Utility Vegetation Management Final Report, concludes that although FirstEnergy s vegetation management practices are within common or average industry practices, those common industry practices need significant improvement to assure greater transmission reliability. b The report further recommends that strict regulatory oversight and support will be required for utilities to improve and sustain needed improvements in their vegetation management programs. NERC has no standards or requirements for vegetation management or transmission right-of-way clearances, nor for the determination of line ratings. U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 59

16 Cause 3 Tree Trimming was no wind cooling of the conductor and the line would not sag excessively. The investigation team examined the relay data for this trip, identified the geographic location of the fault, and determined that the relay data match the classic signature pattern for a tree/line short circuit to ground fault. The field team found the remains of trees and brush at the fault location determined from the relay data. At this location, conductor height measured 46 feet 7 inches (14.20 meters), while the height of the felled tree measured 42 feet (12.80 meters); however, portions of the tree had been removed from the site. This means that while it is difficult to determine the exact height of the line contact, the measured height is a minimum and the actual contact was likely 3 to 4 feet (0.9 to 1.2 meters) higher than estimated here. Burn marks were observed 35 feet 8 inches (10.87 meters) up the tree, and the crown of this tree was at least 6 feet (1.83 meters) taller than the observed burn marks. The tree showed evidence of fault current damage. 24 When the Harding-Chamberlin line locked out, the loss of this 345-kV path caused the remaining three southern 345-kV lines into Cleveland to pick up more load, with Hanna-Juniper picking up the most. The Harding-Chamberlin outage also caused more power to flow through the underlying 138-kV system. Cause 2 Situational Awareness MISO did not discover that Harding-Chamberlin had tripped until after the blackout, when MISO reviewed the breaker operation log that evening. FE indicates that it discovered the line was out while investigating system conditions in response to MISO s call at 15:36 EDT, when MISO told FE that MISO s flowgate monitoring tool showed a Star-Juniper line overload following a contingency loss of Hanna-Juniper; 25 however, the investigation team has found no evidence within the control room logs or transcripts to show that FE knew of the Harding- Chamberlin line failure until after the blackout. Recommendations 16, page 154; 27, page 162 Recommendation 22, page 159 Cause 4 Harding-Chamberlin was not one of the flowgates that MISO monitored RC Diagnostic Support as a key transmission loca- tion, so the reliability coordinator was unaware when FE s first 345-kV line failed. Although MISO received Figure 5.8. Harding-Chamberlin 345-kV Line SCADA input of the line s status change, this was presented to MISO operators as breaker status changes rather than a line failure. Because their EMS system topology processor had not yet been linked to recognize line failures, it did not connect the breaker information to the loss of a transmission line. Thus, MISO s operators did not recognize the Harding-Chamberlin trip as a significant contingency event and could not advise FE regarding the event or its consequences. Further, without its state estimator and associated contingency analyses, MISO was unable to identify potential overloads that would occur due to various line or equipment outages. Accordingly, when the Harding-Chamberlin 345-kV line tripped at 15:05 EDT, the state estimator did not produce results and could not predict an overload if the Hanna- Juniper 345-kV line were to fail. 3C) FE s Hanna-Juniper 345-kV Line Tripped: 15:32 EDT Cause 3 Tree Trimming Recommendation 30, page 163 At 15:32:03 EDT the Hanna- Juniper line (Figure 5.9) tripped and locked out. A tree-trimming crew was working nearby and observed the tree/line contact. The tree contact occurred on the south phase, which is lower than the center phase due to construction design. Although little evidence remained of the tree during the field team s visit in October, the team observed a tree stump 14 inches (35.5 cm) in diameter at its ground line and talked to an individual who witnessed the contact on August Photographs clearly indicate that the tree was of excessive height (Figure 5.10). Surrounding trees were 18 inches (45.7 cm) in diameter at ground line and 60 feet (18.3 meters) in 60 U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations

17 height (not near lines). Other sites at this location had numerous (at least 20) trees in this rightof-way. Hanna-Juniper was loaded at 88% of its normal and emergency rating when it tripped. With this line open, over 1,200 MVA of power flow had to find a new path to reach its load in Cleveland. Loading on the remaining two 345-kV lines increased, with Star-Juniper taking the bulk of the power. This caused Star-South Canton s loading to rise above its normal but within its emergency rating and pushed more power onto the 138-kV system. Flows west into Michigan decreased slightly and voltages declined somewhat in the Cleveland area. Figure 5.9. Hanna-Juniper 345-kV Line Why Did So Many Tree-to-Line Contacts Happen on August 14? Tree-to-line contacts and resulting transmission outages are not unusual in the summer across much of North America. The phenomenon occurs because of a combination of events occurring particularly in late summer: Most tree growth occurs during the spring and summer months, so the later in the summer the taller the tree and the greater its potential to contact a nearby transmission line. As temperatures increase, customers use more air conditioning and load levels increase. Higher load levels increase flows on the transmission system, causing greater demands for both active power (MW) and reactive power (MVAr). Higher flow on a transmission line causes the line to heat up, and the hot line sags lower because the hot conductor metal expands. Most emergency line ratings are set to limit conductors internal temperatures to no more than 100 C (212 F). As temperatures increase, ambient air temperatures provide less cooling for loaded transmission lines. Wind flows cool transmission lines by increasing the airflow of moving air across the line. On August 14 wind speeds at the Ohio Akron-Fulton airport averaged 5 knots (1.5 m/sec) at around 14:00 EDT, but by 15:00 EDT wind speeds had fallen to 2 knots (0.6 m/sec) the wind speed commonly assumed in conductor design or lower. With lower winds, the lines sagged further and closer to any tree limbs near the lines. This combination of events on August 14 across much of Ohio and Indiana caused transmission lines to heat and sag. If a tree had grown into a power line s designed clearance area, then a tree/line contact was more likely, though not inevitable. An outage on one line would increase power flows on related lines, causing them to be loaded higher, heat further, and sag lower. U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations 61

18 3D) AEP and PJM Begin Arranging a TLR for Star-South Canton: 15:35 EDT Cause 4 RC Diagnostic Support Because its alarm system was not working, FE was not aware of the Harding-Chamberlin or Hanna- Juniper line trips. However, once MISO manually updated the state estimator model for the Stuart-Atlanta 345-kV line outage, the software successfully completed a state estimation and contingency analysis at 15:41 EDT. But this left a 36 minute period, from 15:05 EDT to 15:41 EDT, during which MISO did not recognize the consequences of the Hanna-Juniper loss, and FE operators knew neither of the line s loss nor its consequences. PJM and AEP recognized the overload on Star-South Canton, but had not expected it because their earlier contingency analysis did not examine enough lines within the FE system to foresee this result of the Hanna- Juniper contingency on top of the Harding- Chamberlin outage. After AEP recognized the Star-South Canton overload, at 15:35 EDT AEP asked PJM to begin developing a 350 MW TLR to mitigate it. The TLR was to relieve the actual overload above normal rating then occurring on Star-South Canton, and prevent an overload above emergency rating on Figure Cause of the Hanna-Juniper Line Loss This August 14 photo shows the tree that caused the loss of the Hanna-Juniper line (tallest tree in photo). Other 345-kV conductors and shield wires can be seen in the background. Photo by Nelson Tree. Handling Emergencies by Shedding Load and Arranging TLRs Transmission loading problems. Problems such as contingent overloads of normal ratings are typically handled by arranging Transmission Loading Relief (TLR) measures, which in most cases take effect as a schedule change 30 to 60 minutes after they are issued. Apart from a TLR level 6, TLRs are intended as a tool to prevent the system from being operated in an unreliable state, a and are not applicable in real-time emergency situations because it takes too long to implement reductions. Actual overloads and violations of stability limits need to be handled immediately under TLR level 4 or 6 by redispatching generation, system reconfiguration or tripping load. The dispatchers at FE, MISO and other control areas or reliability coordinators have authority and under NERC operating policies, responsibility to take such action, but the occasion to do so is relatively rare. Lesser TLRs reduce scheduled transactions non-firm first, then pro-rata between firm transactions, including flows that serve native load. When pre-contingent conditions are not solved with TLR levels 3 and 5, or conditions reach actual overloading or surpass stability limits, operators must use emergency generation redispatch and/or load-shedding under TLR level 6 to return to a secure state. After a secure state is reached, TLR level 3 and/or 5 can be initiated to relieve the emergency generation redispatch or load-shedding activation. System operators and reliability coordinators, by NERC policy, have the responsibility and the authority to take actions up to and including emergency generation redispatch and shedding firm load to preserve system security. On August 14, because they either did not know or understand enough about system conditions at the time, system operators at FE, MISO, PJM, or AEP did not call for emergency actions. Use of automatic procedures in voltage-related emergencies. There are few automatic safety nets in place in northern Ohio except for underfrequency load-shedding in some locations. In some utility systems in the U.S. Northeast, Ontario, and parts of the Western Interconnection, special protection systems or remedial action schemes, such as under-voltage loadshedding are used to shed load under defined severe contingency conditions similar to those that occurred in northern Ohio on August 14. a Northern MAPP/Northwestern Ontario Disturbance-June 25, 1998, NERC 1998 Disturbance Report, page U.S.-Canada Power System Outage Task Force August 14th Blackout: Causes and Recommendations