As is typical with any large satellite network, not a month goes by without at least one of Intelsat’s more than 50 satellites having some sort of issue with its connection to the terrestrial antenna network. Often it is minor signal interference that can be cleared up in minutes by a technician at one of the company’s network operations centers. Occasionally, the problem is a dropped link that needs to be restored, or a piece of equipment on the ground that has failed and has to be repaired or replaced.
The key to quickly overcoming the little and the big satellite service issues is having a restoration plan and executing on it. As the expression goes, “Hope for the best, but plan for the worst.”
A textbook example of how to react quickly and follow such a plan occurred last summer, when a wildfire in Washington State burned through the terrestrial cables serving a teleport and knocked out commercial satellite service to the U.S. Navy’s Pacific Fleet.
The fire, in July 2014, was the largest in the state’s recorded history, destroying some 300 homes not far from the USEI teleport in Brewster. The fire came within a mile from the 54-antenna complex that is the primary connection for the Navy’s Commercial Broadband Satellite Communications Program (CBSP) services. After the fire burned the fiber connections, cell phones were the only connection the USEI engineers had to the outside world. They used them to quickly notify David Wilson and Gabriel Clark at the Intelsat General CBSP Secure Control Center (CSCC).
Wilson and Clark immediately issued a “Code Red Alert” to the CBSP Management Team, which consisted of Intelsat General and the other dozen companies involved in providing the CBSP service. As the first to respond, Luis Nakpil of Intelsat General and Mike DiLandro of SES immediately initiated a response plan in accordance with the team’s established procedures.
Members of the team assembled in the CBSP Situation Room at Intelsat General’s offices in Bethesda, MD. They immediately went to work to identify all the affected customers and implement a recovery solution.
Recovery of services impacted by the loss of connectivity proved to be a complex technical challenge. The team had to quickly locate candidate teleports and antennas available to restore both C-Band and Ku-Band services to the ships, which were in the midst of a large-scale, multi-national exercise.
In accordance with a restoration plan, the rapid recovery solution involved transferring services to teleports in New Jersey and Hawaii, as well as adding several new satellites to the CBSP network. In New Jersey, the solution required a simple repointing of an antenna to the AMC-9 satellite.
In Hawaii, things were more challenging. One of the antennas needed for restoration of service had been disassembled in preparation for a scheduled overhaul and upgrade. As such, this antenna had to be reassembled and tested to support the transition. Finally, there was another other antenna with a massive 19.2 meter dish that could be used, but belonged to a third party. Engineers needed to contact the company for authorization to repoint it, and make other technical modifications for restoring service.
Within five hours, the team restored service to the first ship with the others restored shortly thereafter. In less than 15 hours, restoration services and link budgets were provided to the Navy – in substantially less time than the threshold allowed and within the bounds of the CBSP contract.
The reason service could be restored so quickly is that the CBSP team had an emergency plan, and then followed the plan with the disaster struck. Happily, such plans rarely need to be taken off the shelf. And when they are implemented, everyone is relieved that they work according to plan!