Nashville's Christmas bombing decimated the 911 network. 2 years later, upgrades are ongoing.

The building had defenses against the blast. It was prepared for the water. Teams had backup power in case of outages.

But they weren't prepared for the crime scene tape.

A stark lesson from the 2020 Christmas Day bombing in downtown Nashville was the need to overhaul redundancy measures in Middle Tennessee's emergency communications system.

The Second Avenue explosion hit critical infrastructure inside the AT&T facility. In the hours after the explosion, The Tennessean, part of the USA TODAY Network, connected the dots as reports of outages radiated out from Nashville to as far away as Kentucky and Alabama.

The aftermath of the Nashville Christmas Day explosion: Phone outages across states

Outages began as backup power inside the facility began to fail and lasted for weeks across the multistate area.

In the following days and weeks, the telecom giant scrambled to restore services while maintaining the integrity of an active investigation site teeming with federal agents. Details of the company's response are laid out in a report compiled by Nashville Mayor John Cooper's Special Bombing Review Commission and released in March.

There were backup generators built to run for hours in case of an outage, but the batteries ran out of power as they froze over in the December weather because of water used to douse fires. Technicians stood by, ready but unable to fix the problem as law enforcement secured and cleared the scene.

No one was killed in the bombing, but crews still had to search structures for days for evidence and any missed victims.

When communications techs could reenter, service was at first only intermittently restored.

Connection for 911 lines damaged

Telecom techs had to balance melting ice with damaging steam, and cycled generators and servers on and off. 911 lines would suddenly reconnect then disconnect again across the region.

It was a wake-up call, a sudden and brutal reminder of vulnerabilities in a trusted network used by both public and private users.

"We restored service as quickly as possible following the Christmas Day bombing in Nashville," a spokesperson for AT&T told The Tennessean in a statement emailed in response to several questions. "Our restoration was delayed while law enforcement processed a difficult crime scene, and unfortunately, caused delays in restoring services."

Nashville's emergency communications team spotted the issue soon after the explosion, and the center was able to quickly pivot to backup pathways when it became clear connectivity was about to drop.

It wasn't perfect, but it was a lot better than several surrounding jurisdictions caught with no 911 systems without warning.

That wasn't just luck, Steve Martini, Nashville's emergency communications director, told The Tennessean. It was in part a testament to his team.

"We have everybody empowered to do their job and do it quickly without asking permission to do it. They trust themselves," he said.

But even as he detailed an ongoing, three-pronged approach to building new redundancies into the region's emergency communications, Martini was unwilling to call the damaged system a total failure.

Similar setups withstood hurricanes and other severe weather across the country, he said. Still, it's vital to him that lessons are implemented.

"That's always a danger — that there's potential to become complacent as weeks turn into months and months turns to a year or two years after an incident like that, that we take the foot off the gas pedal and we don't pursue a solution maybe as eager we would within the first 90 days of the incident," he said.

Setting up new systems for emergency calls, other critical information

Nashville is collaborating with Williamson County to get a landline system up and running using fiber cables to transmit not only emergency calls but other critical information in crises.

No one thinks about getting the city's payroll up and running when the lights go out, he said, but they should.

"Police, fire, EMS — somebody's got to pay those folks their paychecks while they're working those long hours and crazy weeks and trying to keep them safe," he said.

Martini's team has also invested in an internet-based dispatch backup that would allow them to use their computer system on Wi-Fi sent through the cable from Williamson County or any other source if the usual network goes down. During the bombing aftermath, dispatchers at times were using pen and paper, radios and a pile of cell phones on multiple networks to keep things running.

AT&T, too, has made strides on building in more redundancies, the company said. They've worked on improving notification policies with the Tennessee Emergency Communications Board and local agencies, and enhanced power and network resiliency at the Nashville central office.

The company is also doubling down and expanding its growing FirstNet initiative across the state. The auxiliary system focuses on network access for law enforcement and first responders, especially in emergencies where standard networks fail.

The company is tying FirstNet to the emergency call system to detect 911 disruptions and reroute the calls over the separate lines if needed.

The new redundancies are already deep into testing phases and should be ready to be online sometime next year, Martini said.

"Every disaster is a local disaster first," Martini said.

Reach reporter Mariah Timms at mtimms@tennessean.com or 615-259-8344 and on Twitter @MariahTimms.

This article originally appeared on USA TODAY: How Nashville Christmas bombing prompted changes to 911 network