Quality Failures Can Leave You In the Dark

July 13, 2019, a massive power outage plunged a significant portion of New York City into darkness. The outage began Saturday just before 7:00 pm. 73,000 people were in the dark until just after midnight. 

Graphic of a frontline worker fixing a power line

A five-hour power outage may seem relatively insignificant. It certainly pales in comparison to some other more famous power outages. For example, on March 31, 2015, 70 million people in Turkey were without power for more than seven hours. On July 31, 2012, 670 million people in India lost power for between two and eight hours. The famous North American blackout on August 14, 2003 impacted significant portions of the northwestern U.S. and Canada, with more than 50 million people in the dark for between 16-72 hours in the USA and up to 192 hours in Canada. (Veloza and Santamaria 2016)  

New York, however, is one of the world’s most significant metropolitan centres, the hub for a financial network that spans the globe. When a location like New York loses critical infrastructure, the impacts cascade and resonate across financial, transportation, and business systems. 

Power grids are highly reliable. Yet they are also vulnerable to small events that rapidly grow to become perilous. The North American Electric Reliability Corporation (NERC) defines a cascading failure as “the uncontrolled successive loss of system elements triggered by an incident at any location” (Guo et al 2017) Cascading failures are frequently found at the heart of most major blackout scenarios. According to Larson et al (2007), most large blackouts begin when a series of small problems occur that either aren’t noticed or are dismissed as not significant enough to cause cascading failures. 

This pattern was in play on July 15, 2019. Con Edison, the utility company that provides power to New York City, released a statement blaming the blackout on the failure of their relay protection system to isolate a damaged 13,000 volt cable at West 64th Street and West End Avenue.  

A damaged cable is a common occurrence in a large power grid like New York’s. In the initial hours after the blackout, Con Edison was reluctant to lay the blame on something so routine. However, the company subsequently determined that the relay protection did not operate as designed, and both the primary and backup relay protection systems failed to isolate the cable. That relatively small failure cascaded throughout the system and cut off power to six neighborhood networks in the city. According to Con Edison, they will continue investigating why multiple redundant systems failed to meet design specifications. 

While there were no significant injuries or hospitalizations, there are important lessons to learn from this event. One is the cost of downtime in a complex system. While there are no official figures on the cost of the blackout in terms of downtime and lost productivity, the costs in a major business center are significant, even on a weekend when the city’s financial markets are closed.  

Root Cause Analysis

Second, and perhaps most importantly, is the requirement for a rigorous root cause analysis to determine how this happened and to prevent its reoccurrence. Analytical tools such as Failure Mode and Effects Analysis during the design phase (FMEA) are commonly used throughout the power industry to anticipate and prevent the risk of equipment failure like what was observed with the relay protection equipment. Why controls for prevention and detection failed to uncover the problems that led to the New York City blackout, or whether the root cause is related to the design of one or more components, will no doubt require significant effort. 

Sources 

Guo, H., Zheng, C., Iu, H. H. C., & Fernando, T. (2017). A critical review of cascading failure analysis and modeling of power system. Renewable and Sustainable Energy Reviews, 80, 9-22. 

Larsson, J. E., Öhman, B., & Calzada, A. (2007). Real-time root cause analysis for power grids. In Security and Reliability of Electric Power Systems, CIGRE Regional Meeting, Tallinn, Estonia. 

Veloza, O. P., & Santamaria, F. (2016). Analysis of major blackouts from 2003 to 2015: Classification of incidents and review of main causes. The Electricity Journal, 29(7), 42-49. 

One thought on “Quality Failures Can Leave You In the Dark

  1. I totally agree with the concept but we should start earlier by conducting an efficient risk assessment to get the control measures in place and validate it. FMEA is pretty good in that. 🤞

Leave a Reply

Your email address will not be published. Required fields are marked *