Welcome to DU!
The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards.
Join the community:
Create a free account
Support DU (and get rid of ads!):
Become a Star Member
Latest Breaking News
General Discussion
The DU Lounge
All Forums
Issue Forums
Culture Forums
Alliance Forums
Region Forums
Support Forums
Help & Search
16 Years Ago Today; The Northeast Blackout of 2003
https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
This image shows states and provinces that experienced power outages. Not all areas within these political boundaries were affected.
The Northeast blackout of 2003 was a widespread power outage throughout parts of the Northeastern and Midwestern United States, and the Canadian province of Ontario on August 14–28, 2003, beginning just after 4:10 p.m. EDT.
Some power was restored by 11 p.m. Most did not get their power back until two days later. In other areas, it took nearly a week or two for power to be restored. At the time, it was the world's second most widespread blackout in history, after the 1999 Southern Brazil blackout. The outage, which was much more widespread than the Northeast blackout of 1965, affected an estimated 10 million people in southern and central Ontario, and 45 million people in eight U.S. states.
The blackout's primary cause was a software bug in the alarm system at the control room of FirstEnergy, an Akron, Ohio–based company, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage. What should have been a manageable local blackout cascaded into collapse of the entire Northeast region.
Immediate impact
According to the New York Independent System Operator (NYISO) – the ISO responsible for managing the New York state power grid – a 3,500 megawatt power surge (towards Ontario) affected the transmission grid at 4:10:39 p.m. EDT.
For the next 30 minutes, to 4:40 p.m. EDT, outages were reported in parts of Ohio, New York, Michigan, and New Jersey: Cleveland, Akron, Toledo, New York City, Westchester, Orange and Rockland counties, Baltimore, Rochester, Syracuse, Binghamton, Albany, Detroit, and parts of New Jersey, including the city of Newark.
This was followed by outages in other areas initially unaffected, including all of New York City, portions of southern New York state, New Jersey, Vermont, Connecticut, as well as most of the province of Ontario, including Toronto. Eventually, a large, somewhat triangular area bounded by Lansing, Michigan, Sault Ste. Marie, Ontario, the shore of James Bay, Ottawa, New York, and Toledo was left without power.
According to the official analysis of the blackout prepared by the US and Canadian governments, more than 508 generating units at 265 power plants shut down during the outage. In the minutes before the event, the NYISO-managed power system was carrying 28,700 MW of load. At the height of the outage, the load had dropped to 5,716 MW, a loss of 80%.
Essential services remained in operation in some of these areas. In others, backup generation systems failed. Telephone networks generally remained operational, but the increased demand triggered by the blackout left many circuits overloaded. Water systems in several cities lost pressure, forcing boil-water advisories to be put into effect. Cellular service was interrupted as mobile networks were overloaded with the increase in volume of calls. Major cellular network providers lost service until power was restored.[citation needed] Television and radio stations remained on the air, with the help of backup generators, although some stations were knocked off the air for periods ranging from several hours to the length of the entire blackout.
It was a hot day (over 31 °C, or 88 °F) in much of the affected region, and the heat played a role in the initial event that triggered the wider power outage. The high ambient temperature increased energy demand, as people across the region turned on fans and air conditioning. This caused the power lines to sag as higher currents heated the lines.
In areas where power remained off after nightfall, the Milky Way and orbiting artificial satellites became visible to the naked eye in metropolitan areas where they cannot ordinarily be seen due to the effects of particulate air and light pollution.
Most of the Amtrak Northeast Corridor service was interrupted, as it uses electric locomotives; electrified commuter railways also shut down. Via Rail in Canada was able to continue most of its service. All airports in the affected area closed immediately, so there were no take-offs, and incoming flights had to be diverted to airports with power. U.S. financial markets, which are completely power-dependent, were also shut down. The reliability of the electrical grid was called into question, meaning it was deficient, of less value, and required substantial investment to repair its shortcomings.
<snip>
Causes
Satellite imagery of the Northeastern United States taken before and during the blackout
Background
The load on any power network must be immediately matched by its supply and its ability to transmit that power. Any overload of a power line or generator can cause costly damage, so the affected device is disconnected from the network if an overload is detected.
The electrical resistance of a power line causes it to produce more heat as the current it carries increases. If this heat is not sufficiently dissipated, the metal conductor in the line will expand and lengthen, so that it sags between supporting structures. If the line sags too low, a flash over to nearby objects (such as trees) may occur, causing a transient increase in current. Automatic protective relays detect the excessively high current and quickly disconnect the line, with the load previously carried by the line transferred to other lines. If the other lines do not have enough spare capacity to accommodate the extra current, their overload protection will react as well, causing a cascading failure.
System operators are responsible for ensuring that power supply and loads remain balanced, and for keeping the system within safe operational limits such that no single fault can cause the system to fail. After a failure affecting their system, operators must obtain more power from generators or other regions or "shed load" (meaning to intentionally cut power or reduce voltage to a given area) until they can be sure that the worst remaining possible failure anywhere in the system will not cause a system collapse. In an emergency, they are expected to immediately shed load as required to bring the system into balance.
To assist the operators there are computer systems, with backups, which issue alarms when there are faults in the transmission or generation system. Power flow modeling tools let them analyze the state of their network, predict whether any parts of it may be overloaded, and predict the worst possible failure remaining, so that they can change the distribution of generation or reconfigure the transmission system to prevent a failure should this situation occur. If the computer systems and their backups fail, the operators are required to monitor the grid manually, instead of relying on computer alerts. If they cannot interpret the state of the power grid in such an event, they follow a contingency plan, contacting other plant and grid operators by telephone if necessary. If there is a failure, they are also required to notify adjacent areas which may be affected, so those can predict the possible effects on their own systems.
Investigation efforts
A joint federal task force was formed by the governments of Canada and the U.S. to oversee the investigation and report directly to Ottawa and Washington. The task force was led by then-Canadian Natural Resource Minister Herb Dhaliwal and U.S. Energy Secretary Spencer Abraham.
In addition to determining the initial cause of the cascading failure, the investigation of the incident also included an examination of the failure of safeguards designed to prevent a repetition of the Northeast blackout of 1965. The North American Electric Reliability Corporation, a joint Canada-U.S. council, is responsible for dealing with these issues.
On November 19, 2003, Abraham said his department would not seek to punish FirstEnergy Corp for its role in the blackout because current U.S. law does not require electric reliability standards. Abraham stated, "The absence of enforceable reliability standards creates a situation in which there are limits in terms of federal level punishment."
Findings
In February 2004, the U.S.-Canada Power System Outage Task Force released their final report, placing the causes of the blackout into four groups:
Computer failure
A software bug known as a race condition existed in General Electric Energy's Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy's control room alarm system for over an hour. System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state.[9][10]
Unprocessed events queued up after the alarm system failure and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were automatically transferred to the backup server, which itself failed at 14:54. The server failures slowed the screen refresh rate of the operators' computer consoles from 1–3 seconds to 59 seconds per screen. The lack of alarms led operators to dismiss a call from American Electric Power about the tripping and reclosure of a 345 kV shared line in northeast Ohio. But by 15:42, after the control room itself lost power, control room operators informed technical support (who were already troubleshooting the issue) of the alarm system problem.
</snip>
This image shows states and provinces that experienced power outages. Not all areas within these political boundaries were affected.
The Northeast blackout of 2003 was a widespread power outage throughout parts of the Northeastern and Midwestern United States, and the Canadian province of Ontario on August 14–28, 2003, beginning just after 4:10 p.m. EDT.
Some power was restored by 11 p.m. Most did not get their power back until two days later. In other areas, it took nearly a week or two for power to be restored. At the time, it was the world's second most widespread blackout in history, after the 1999 Southern Brazil blackout. The outage, which was much more widespread than the Northeast blackout of 1965, affected an estimated 10 million people in southern and central Ontario, and 45 million people in eight U.S. states.
The blackout's primary cause was a software bug in the alarm system at the control room of FirstEnergy, an Akron, Ohio–based company, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage. What should have been a manageable local blackout cascaded into collapse of the entire Northeast region.
Immediate impact
According to the New York Independent System Operator (NYISO) – the ISO responsible for managing the New York state power grid – a 3,500 megawatt power surge (towards Ontario) affected the transmission grid at 4:10:39 p.m. EDT.
For the next 30 minutes, to 4:40 p.m. EDT, outages were reported in parts of Ohio, New York, Michigan, and New Jersey: Cleveland, Akron, Toledo, New York City, Westchester, Orange and Rockland counties, Baltimore, Rochester, Syracuse, Binghamton, Albany, Detroit, and parts of New Jersey, including the city of Newark.
This was followed by outages in other areas initially unaffected, including all of New York City, portions of southern New York state, New Jersey, Vermont, Connecticut, as well as most of the province of Ontario, including Toronto. Eventually, a large, somewhat triangular area bounded by Lansing, Michigan, Sault Ste. Marie, Ontario, the shore of James Bay, Ottawa, New York, and Toledo was left without power.
According to the official analysis of the blackout prepared by the US and Canadian governments, more than 508 generating units at 265 power plants shut down during the outage. In the minutes before the event, the NYISO-managed power system was carrying 28,700 MW of load. At the height of the outage, the load had dropped to 5,716 MW, a loss of 80%.
Essential services remained in operation in some of these areas. In others, backup generation systems failed. Telephone networks generally remained operational, but the increased demand triggered by the blackout left many circuits overloaded. Water systems in several cities lost pressure, forcing boil-water advisories to be put into effect. Cellular service was interrupted as mobile networks were overloaded with the increase in volume of calls. Major cellular network providers lost service until power was restored.[citation needed] Television and radio stations remained on the air, with the help of backup generators, although some stations were knocked off the air for periods ranging from several hours to the length of the entire blackout.
It was a hot day (over 31 °C, or 88 °F) in much of the affected region, and the heat played a role in the initial event that triggered the wider power outage. The high ambient temperature increased energy demand, as people across the region turned on fans and air conditioning. This caused the power lines to sag as higher currents heated the lines.
In areas where power remained off after nightfall, the Milky Way and orbiting artificial satellites became visible to the naked eye in metropolitan areas where they cannot ordinarily be seen due to the effects of particulate air and light pollution.
Most of the Amtrak Northeast Corridor service was interrupted, as it uses electric locomotives; electrified commuter railways also shut down. Via Rail in Canada was able to continue most of its service. All airports in the affected area closed immediately, so there were no take-offs, and incoming flights had to be diverted to airports with power. U.S. financial markets, which are completely power-dependent, were also shut down. The reliability of the electrical grid was called into question, meaning it was deficient, of less value, and required substantial investment to repair its shortcomings.
<snip>
Causes
Satellite imagery of the Northeastern United States taken before and during the blackout
Background
The load on any power network must be immediately matched by its supply and its ability to transmit that power. Any overload of a power line or generator can cause costly damage, so the affected device is disconnected from the network if an overload is detected.
The electrical resistance of a power line causes it to produce more heat as the current it carries increases. If this heat is not sufficiently dissipated, the metal conductor in the line will expand and lengthen, so that it sags between supporting structures. If the line sags too low, a flash over to nearby objects (such as trees) may occur, causing a transient increase in current. Automatic protective relays detect the excessively high current and quickly disconnect the line, with the load previously carried by the line transferred to other lines. If the other lines do not have enough spare capacity to accommodate the extra current, their overload protection will react as well, causing a cascading failure.
System operators are responsible for ensuring that power supply and loads remain balanced, and for keeping the system within safe operational limits such that no single fault can cause the system to fail. After a failure affecting their system, operators must obtain more power from generators or other regions or "shed load" (meaning to intentionally cut power or reduce voltage to a given area) until they can be sure that the worst remaining possible failure anywhere in the system will not cause a system collapse. In an emergency, they are expected to immediately shed load as required to bring the system into balance.
To assist the operators there are computer systems, with backups, which issue alarms when there are faults in the transmission or generation system. Power flow modeling tools let them analyze the state of their network, predict whether any parts of it may be overloaded, and predict the worst possible failure remaining, so that they can change the distribution of generation or reconfigure the transmission system to prevent a failure should this situation occur. If the computer systems and their backups fail, the operators are required to monitor the grid manually, instead of relying on computer alerts. If they cannot interpret the state of the power grid in such an event, they follow a contingency plan, contacting other plant and grid operators by telephone if necessary. If there is a failure, they are also required to notify adjacent areas which may be affected, so those can predict the possible effects on their own systems.
Investigation efforts
A joint federal task force was formed by the governments of Canada and the U.S. to oversee the investigation and report directly to Ottawa and Washington. The task force was led by then-Canadian Natural Resource Minister Herb Dhaliwal and U.S. Energy Secretary Spencer Abraham.
In addition to determining the initial cause of the cascading failure, the investigation of the incident also included an examination of the failure of safeguards designed to prevent a repetition of the Northeast blackout of 1965. The North American Electric Reliability Corporation, a joint Canada-U.S. council, is responsible for dealing with these issues.
On November 19, 2003, Abraham said his department would not seek to punish FirstEnergy Corp for its role in the blackout because current U.S. law does not require electric reliability standards. Abraham stated, "The absence of enforceable reliability standards creates a situation in which there are limits in terms of federal level punishment."
Findings
In February 2004, the U.S.-Canada Power System Outage Task Force released their final report, placing the causes of the blackout into four groups:
FirstEnergy (FE) and its reliability council "failed to assess and understand the inadequacies of FE's system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria."
FirstEnergy "did not recognize or understand the deteriorating condition of its system."
FirstEnergy "failed to manage adequately tree growth in its transmission rights-of-way."
Finally, the "failure of the interconnected grid's reliability organizations to provide effective real-time diagnostic support."
The report states that a generating plant in Eastlake, Ohio, a suburb northeast of Cleveland, went offline amid high electrical demand, putting a strain on high-voltage power lines (located in Walton Hills, Ohio, a southeast suburb of Cleveland) which later went out of service when they came in contact with "overgrown trees". This trip caused load to transfer to other transmission lines, which were not able to bear the load, tripping their breakers. Once these multiple trips occurred, many generators suddenly lost parts of their loads, so they accelerated out of phase with the grid at different rates, and tripped out to prevent damage. The cascading effect that resulted ultimately forced the shutdown of more than 100 power plants.
Computer failure
A software bug known as a race condition existed in General Electric Energy's Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy's control room alarm system for over an hour. System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state.[9][10]
Unprocessed events queued up after the alarm system failure and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were automatically transferred to the backup server, which itself failed at 14:54. The server failures slowed the screen refresh rate of the operators' computer consoles from 1–3 seconds to 59 seconds per screen. The lack of alarms led operators to dismiss a call from American Electric Power about the tripping and reclosure of a 345 kV shared line in northeast Ohio. But by 15:42, after the control room itself lost power, control room operators informed technical support (who were already troubleshooting the issue) of the alarm system problem.
</snip>
That afternoon, I was driving on the NYS Thruway between Schenectady and Albany and received a call from my office asking if the lights were out where I was. Only when I reached Exit 24 did I see any sign of the power failure.
2 replies
= new reply since forum marked as read
= new reply since forum marked as read
It was mostly a daylight event. IIRC, the lights came back on in the Albany area before nightfall. 