Ookla recently published an interesting article that emphasizes what I have been telling folks for a long time: the consolidation of Internet networks and services has increased the risk of more significant outages.
Not that many years ago, telephone and broadband networks were structured in such a way that most outages were local events. A fiber cut might disrupt service to a neighborhood; an electronics failure might affect a larger area, but for the most part, Internet outages were contained within a discrete and local area.
There were exceptions. Rural areas have been susceptible to fiber cuts in the fiber that provides the Internet backbone. Years ago, I worked with Cook County, Minnesota, which would lose voice and broadband every time there was a cut in the single fiber between Minneapolis and northern Minnesota that supported the area. A public-private partnership was established to develop the THOR network, aiming to address backhaul failures in a significant portion of southeastern Colorado.
As the article points out, this has all changed because network operators have consolidated and interconnected networks across large geographic areas. Ookla says that the new phenomenon of large-scale outages is a direct result of digital transformation. As carriers, companies, and governments have grown increasingly reliant on cloud services, managed providers, and interconnected networks, they now must contend with outages that can cascade from a local issue to a regional or even national problem.
The article examines the recent power outage in Spain and Portugal, which quickly escalated from a local incident to a widespread power outage across much of the Iberian Peninsula. Ookla points out that in today’s world, there is not that much difference between outages of a power grid, a cellular network, or a fiber network.
The article notes that outages can cascade much faster than anybody expects. The difference between a temporary disruption and a system-wide crisis depends on how quickly the network operators can recognize and analyze the causes of a problem.
Five Steps To De-escalate Disruptions
Ookla says there are five key steps needed to keep disruptions from escalating:
- Detection: Identify the first signs of trouble across multiple data sources, including outage reports and operator dashboards.
- Attribution: Diagnose the root cause of the problem, whether it’s an internal software bug, a fiber cut, or a regional power failure.
- Communication: Share timely and accurate information with stakeholders and the public to minimize confusion.
- Remediation: Act quickly to contain damage, restore critical services, and prevent cascading failures.
- Learning: Capture lessons from each event and feed them back into playbooks, exercises, and long-term resilience planning.
Every major network outage is likely due to network operators failing at one of the early steps of this process.
Ookla believes that the local reaction within the first hour can significantly impact the extent and duration of an outage. There was one power company in Iberia that was able to isolate itself from the cascading shutdown because it was prepared to react quickly.
I wonder how many local ISPs are ready to respond rapidly to problems originating outside their local network?
Doug Dawson is President of CCG Consulting and writes an industry blog for small and medium carriers.
Adapted from the original post, which first appeared on the Pots and Pans Blog.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of the Internet Society.