On a ship at sea, fire prevention is a major concern. So it is ironic that a fire, not at sea but in a forest 50 miles inland, could knock out satellite services to U.S. Navy ships in the Pacific Ocean. The fire, known as the Carlton Complex wildfire, occurred this past July in Washington State near the USEI teleport in Brewster. The blaze burned for more than two weeks, destroying terrestrial cable connections to the teleport and 300 homes over an area of 400 square miles. It was the largest fire in Washington State’s recorded history.
On-site engineers at the teleport had been monitoring the fire in the late night hours of July 17 when they learned that it had unexpectedly changed direction and was headed directly toward their facility. Just after midnight, as they prepared for the worst at the 54-antenna complex, the fire came within a mile and consumed the terrestrial connectivity into the teleport. As a result, the Navy lost its Commercial Broadband Satellite Communications Program (CBSP) services. USEI engineers were left with cellular telephones as their sole lifeline to the outside world. They quickly notified their counterparts David Wilson and Gabriel Clark at the Intelsat General CBSP Secure Control Center (CSCC), who in turn immediately sent out a “Code Red Alert” to the CBSP Management Team, consisting of Intelsat General and the other dozen companies involved in providing the CBSP service.
Luis Nakpil of Intelsat General and Mike DiLandro of SES were the first to respond, and immediately initiated a response plan in accordance with the team’s established procedures. Soon after, other members of the team arrived on-site in the CBSP Situation Room at Intelsat General’s offices in Bethesda, MD. The team included Bill Flynn of SES and, from Intelsat General, Jonathan Bae, John Rasmussen, Darin Fleming, Jeff Chiocchio, Steve Jackson, Hank Rausch, Dennis Boiter, Todd Cress, Tim Turk, Bill Andrews, James Spaulding and even Hector Vivanco, who called in from his vacation at the World Cup in Brazil. They immediately went to work to identify all the affected customers and to implement a recovery solution.
Recovery of services impacted by the loss of connectivity at Brewster proved to be a complex technical challenge. The team quickly identified candidate teleports and antennas available to restore both C-Band and Ku-Band services to the ships which were in the midst of a large-scale, multi-national exercise. As the situation unfolded, the recovery solution involved transferring services, in accordance with a restoration plan, to teleports in Holmdel, NJ, and Paumalu, HI, in addition to adding several new satellites to the CBSP network.
The Holmdel solution was fairly straightforward and simply required repointing of an antenna to the AMC-9 satellite. However, the Paumalu solution would prove to be more complex and challenging. One of the antennas in Paumalu needed for restoration of service had been disassembled in preparation for a scheduled overhaul and upgrade. This antenna had to be reassembled and tested to support the transition. Finally, one other antenna, with a massive 19.2 meter dish, was required to support the transition. This large antenna belonged to a third party, so engineers contacted the company for authorization to repoint it and make other technical modifications that would help restore service.
The teleport engineers mobilized to rebuild and repoint their antennas beginning at 3 am Hawaii time, which required working in the dark. At the same time, the CBSP customer solutions engineering team went to work on developing link budgets and slot matrices for the CSCC to use when the hardware and other aspects of the technical solution were all in place. In the meantime, the project leaders in the CBSP Situation Room were busy planning and coordinating with external parties and the Navy customer to ensure they were all prepared to reconnect to the network when the earth stations and link budgets were ready to support the transition.
Engineers restored service to the first ship within about five hours, and service to the others soon followed. The team provided all restoration services and link budgets to the Navy in less than 15 hours, allowing shipboard technicians to make an orderly transition to the new services. Services were restored to the Navy in substantially less time than the threshold allowed within the bounds of the CBSP contract.