This is an emergency announcement. At approximately 6:30am today, December 25th, 2020 a bomb was detonated outside of the Nashville, TN AT&T building, causing massive damage to the AT&T building, and surrounding buildings. As of about 11:30am there was a failure of backup systems in the AT&T network, due to damage from this event. Due to the failure of the main systems, and backup systems, there are massive outages at this time. Nashville authorities are still responding to the incident. This incident unfortunately has resulted in at least one death, and three injuries.
This building, the Nashville AT&T building, is a major fiber internet transport for both local and national fiber internet transport. This building also houses some critical AT&T systems. At this time, 12:45pm CST, AT&T is experiencing massive service outages. These outages are affecting ALL AT&T services in some areas, including AT&T TV, AT&T Fiber, AT&T Landline Phones, AT&T Wireless Phones, AT&T LTE, AT&T 5G, FirstNET phone and data.
The FBI and ATF have requested Nashville to shut down part of the electrical grid, and natural gas lines in parts of the city. This means that some systems that are currently online, will probably go offline in the next few hours once battery backups run out of power. They have not released a timeline of when electrical and natural gas service will be restored, we expect it will be some time, possibly a few days.
Sardis Telecom has noticed through network monitoring that about 400 customers of ours are 100% offline, and a few thousand are having intermittent packet loss, or inability to connect to some websites due to the traffic trying to go over network paths that are down.
At this time, there is no estimated time for a full fix. The area in Nashville is an active crime scene, and then AT&T will start repairs. AT&T is actively working to re-route services until then, but the priority is getting FirstNET back online and operational first, followed by business and consumer level services.
We are very glad no one has been reported killed, in this event, an event that appears to be a terrorist event.
Please remain vigilant, and report all suspicious activity to your local authorities.
Statement from AT&T: “This morning’s Nashville bombing damaged AT&T’s regional switching center. Much of the Southeastern US may be impacted at some level including other Internet and communication carriers. AT&T is providing regular updates to all major news services. It’s likely to be a while before all services can be restored.”
Updated Statement from AT&T, 5pm CST on 12/25/2020 – https://about.att.com/pages/disaster_relief/nashville.html
Additional Information from AT&T NOC: The Nashville building (NSVLTNMT) has sustained significant structural damage to the front one-third of the building; the rear two-thirds of the building appears to be intact. Parts of the interior are still inaccessible for safety reasons due to the structural damage sustained. Structural and electrical engineers are en route.
There is standing water in the basement and on every floor. The fire department will assist with pumping water out of basement.
There is no commercial power connectivity to the equipment due to breaker and switch equipment damage. Teams are currently working to de-energize all systems.
Two 850kW generators are en route with an ETA of 21:00 CT. A chiller is also en route with no ETA available at this time. The parking lot adjacent to the building (opposite the damaged side) has been secured to stage the generators. Once the generators arrive, teams will determine if it is safe to wire the generators tonight due to the sustained building damage and water in the building.
In addition, TEMA (Tennessee Emergency Management Agency) is volunteering two 1 megawatt generators to support recovery.
A SATCOLT has been positioned approximately a block and a half from the NSVLTNMT building and is up and taking traffic to aid in communication with those performing restoration. The next update will be issued at approximately 21:00 CT, unless there is a significant change in status.
Previous Status: As of 14:30 CT: There are multiple Battery on Discharge (BOD) conditions in the Nashville, TN Main (NSVLTNMT) Central Office (CO). The conditions observed are related to an explosion in downtown Nashville, that occurred near an AT&T building. The office affected supports several critical services including VOLTE, 3G/4G, TDM Switching, BVOIP services, Uverse/Data Services, 911 Routing, ESINET-911, and certain Enterprise Business Services. This event is affecting Mobility Services, 911 Services, TDM Voice Services, Data services and DirecTV channels in the area. There are seven (7) battery plants on discharge and voltages continue to drop. There are high temperature conditions due to HVAC systems failure and there is also water flooding the basement. Technicians and contractors from ATS, TFS Power and CRE have gained access to the office and are currently assessing the damage and developing service restoration and building repair plans. Additional resources and contractors have been dispatched to the area to assist, as needed. The AT&T National Disaster Recovery team has been engaged and is sending multiple SAT COLTs to the affected area. A working bridge has been established and teams continue to assess the situation and prepare for any restoration efforts that may be necessary. The next update will be at approximately 17:30 CT, unless there is a significant change in status.
Please also note, this is also affecting customers with EPB, Charter, Verizon and more, as this has affected major internet transit routes.