Viodi View – The Bridge Between the Heartland and Silicon Valley

Could Major Telecom Outage in Santa Clara County Been Prevented or Alleviated?

On Thursday April 9th, Internet, cell phone, and land-line phone service were all out of service for customers in parts of Santa Clara, Santa Cruz and San Benito Counties. Even 911 calls weren’t going through. The culprit was a gang of rogues that cut 10 AT&T fiber optic cables in 4 different locations.   The SF Chronicle reports that:

Sabotage attacks knock out phone service
http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/04/10/MNP816VTE6.DTL

The San Jose Mercury weighs in with an article asking:

South Valley phone outage: Is telecom system too vulnerable?
http://www.mercurynews.com/topstories/ci_12111011

"Some of the questions that come up are whether the system has adequate backup capabilities and whether or not the phone companies are adequately protecting and maintaining the equipment they do have," said Mindy Spatt, spokeswoman for The Utility Reform Network, a consumer watchdog group in San Francisco.

The loss of landline, cell phone and Internet service in parts of southern Santa Clara and Santa Cruz counties, as well as areas of San Benito County, was amplified by the fact that multiple carriers were sending traffic on the same fiber-optic cables, which are owned by AT&T.

For example, Verizon said more than 50,000 household landlines were affected as well as an unknown number of cell phone connections.

Cell phones function like miniature radios, communicating with cell phone towers via radio waves. The towers, in turn, communicate with a switching station either over cable or through microwaves.

"If the switch is not working, you are out of business," said Ken Fattlar, a Verizon engineer.  Fattlar said the cell towers in the South Bay are connected to copper cables that feed into AT&T’s fiber lines. When the lines were damaged, the cell towers were not able to communicate with their switching station and could not transfer calls.
—————————————————————————————————————-
Opinion-  Why no early fault detection, cable redundancy and automatic switchover (to another fiber optic cable) when the primary cable was cut?

"Backhoes cut cables all the time," said Bruce Schneier, author of "Schneier on Security," and "Secrets and Lies: Digital Security in a Networked World." "Trees cut cables all the time." Why didn’t AT&T take care to automatically detect the broken cable and switch traffic to an alternate route?

If the previously announced (CLEAR) WiMAX services (Internet and VoIP) were operational in Silicon Valley, it likely would not have been effected, as Clearwire uses microwave backhaul (DragonWave equipment) to reach the ISP or IXC POPs.

So, there is yet another reason to root for WiMAX success- break the monopoly that AT&T and VZ have on U.S. telecommunications by offering customers an alternate path to the Internet and long distance telephony. Additionally, WiMAX could be used as an Internet access backup by corporations needing high availability and  reliabilty.

Author Alan Weissberger

View all posts by Alan Weissberger | Website

Posted

in

by

Alan Weissberger

Tags:

, , , , , ,

Comments

3 responses to “Could Major Telecom Outage in Santa Clara County Been Prevented or Alleviated?”

  1. Goff Avatar
    Goff

    The fact that it took the vandals cutting through 10 fiber lines to knock out only ~50,000 voice lines shows exactly how redundant the system is. I venture that any company that has that many fibers cut is bound to have massive outages. I would also imagine that if you could look at the times that the alarms for the cut fibers started sounding, they would all have been very close to one another suggesting that this was a coordinated attack.

    Also with a microwave backbone couldn’t those frequencies be “jammed” making the WiMax backbone inoperable?

    Reply
  2. Jonathan Wells Avatar
    Jonathan Wells

    Clearly fiber is not as robust as advertised. Competitors lease services to each other, often down the same fiber strands, and even redundant fiber networks often use fibers routed down the same conduits. This lends such networks to relatively easy disruption by malicious attackers.

    Alan is correct that microwave backhaul (as provided by vendors such as Dragonwave) is far more resilient to attacks, as there are no physical “wires” to “cut”, hardware is mounted high on poles, and diversity systems truly are physically separated.

    Goff, microwave backhaul cannot practically be jammed. Unlike cellphones or WiFi which operate on narrow frequency bands, in the US, 6, 11, 15, 18, 23 and 38 GHz are all routinely used for backhaul – a very wide spread of frequencies. Also, microwave backhaul uses high directivity, point to point antennas usually mounted high above the ground, forming narrow focused cone-like antenna patterns across the link. To jam these, the intruder would need a similar high directivity antenna to point towards the antenna he is trying to interfere with to generate enough jamming power to interfere with the required signal (i.e. the jamming source would be isolated by the receive antenna radiation pattern, and so the jammer would need very high power to cause “leakage” into the antenna). Alternatively the jammer would need to get “in the path”, which would be essentially impossible since microwave links are designed to have no obstacles in the line of sight.

    Reply
  3. Alan Weissberger Avatar
    Alan Weissberger

    On April 17th, I attended a Telecom Council wireless infrastructure seminar that talked about using microwave backhaul in support of 3.5G and 4G wireless technologies, including LTE. The participants all felt there was sufficient capacity in the microwave backhaul links to support all of the existing and new technologies.
    See Seminar Agenda at:
    http://guest.cvent.com/EVENTS/Info/Agenda.aspx?e=011e0088-99d0-4fc3-adf3-38427ac4c709

    The problem with fiber failover recovery:
    Truly diverse routes must be found for redundant links and this is rarely the case. As Jonathan points out, “Telco competitors lease services to each other, often down the same fiber strands, and even redundant fiber networks often use fibers routed down the same conduits. This lends such networks to relatively easy disruption by malicious attackers.”

    For many years I worked for transmission equipment vendors implementing fiber fault recovery systems. Bi-directional Line Swithed Rings (BLSR), 1:1 (Automatic Protection Switching (APS), or 1:N APS should be capable of 50ms fail-over for recovery. But ONLY if diverse path routing is used for the redundant fiber links. Evidently, this was not the case for AT&T’s network that experienced the telecom outage.

    It is much harder to jam microwave backhaul links then to cut fiber optic cables. Indeed, jamming microwave backhaul would be very difficult as Jonathan points out. But even if it could be done, the jammer would only be able to jam one end of one link, and not take down a whole network- which was the case in Santa Clara county on April 9th when cell phone, landline phone/fax/modem, and high speed Internet access were all down, as was 911 emergency service.

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.