What was the FAA glitch?

On January 11, 2023, a major computer outage caused a ground stop for all domestic departing flights from U.S. airports, leading to widespread flight delays and cancellations. This disruption was the result of a failure in the Federal Aviation Administration’s (FAA’s) Notice to Air Missions (NOTAM) system. A NOTAM is an advisory alerting pilots and airport personnel regarding important flight information and potential hazards along a flight route.

What is a NOTAM?

A Notice to Air Missions (NOTAM) is a notice containing information essential to personnel concerned with flight operations, but not known far enough in advance to be publicized by other means. NOTAMs contain information on unanticipated or temporary changes to components of, or hazards in, the National Airspace System (NAS). Some examples of information contained in NOTAMs:

  • Construction or maintenance work on airport runways
  • Inoperable navigational aids or lighting systems
  • Weather hazards
  • Military exercises with resulting airspace restrictions
  • Inoperable radio frequencies
  • Closed taxiways

NOTAMs provide pilots with the most current information so they can make informed decisions regarding safety of flight. Before every flight, pilots must check NOTAMs as part of flight planning. The NOTAM system in the U.S. is centralized through the NOTAM System Command Center in Leesburg, Virginia, which is part of the FAA’s Air Traffic Organization.

What happened on January 11, 2023?

In the early morning hours of January 11, 2023, the FAA experienced an outage with its NOTAM system that provides safety information to flight crews. This critical system stopped processing and failed to send updated information, leading the FAA to halt all domestic departures until 9 a.m. EST to allow time to validate the integrity of flight and safety information.

Some key events in the FAA system outage timeline:

  • Around 1 a.m. EST – Contractors working on the NOTAM system caused a database corruption that impacted the pilot briefing system
  • 5:28 a.m. EST – FAA orders a ground stop for all domestic departures until 9 a.m. EST to allow review of system data
  • 7:19 a.m. EST – FAA identifies the cause as a damaged database file
  • 8:50 a.m. EST – FAA begins restarting the system
  • 9 a.m. EST – FAA lifts ground stop

During the outage, a total of 5,483 domestic flights were delayed and 921 were canceled, according to FlightAware data. The NOTAM system continued disseminating information during the outage, but it was incomplete and outdated. The damaged database prevented new information from being conveyed to personnel.

What is the NOTAM system?

The NOTAM system consists of several integrated components that distribute vital aeronautical information to pilots, flight crews, and other users:

  • AIS-R Database – The Aeronautical Information Services Replacement is the database that collects, archives, and disseminates NOTAM information.
  • EFSTS – The Enhanced Flight Service Workstation Terminal System queries the AIS-R database and formats NOTAM data for briefing purposes.
  • PilotWeb – An online portal that provides self-briefing capability, allowing pilots to access NOTAM data.
  • Telephone briefings – Flight service station specialists provide one-on-one telephone briefings to pilots.
  • FAA teleprinter network – Disseminates NOTAM information to air traffic facilities.

This integrated system architecture manages a huge volume of NOTAM data – around 35,000 active NOTAMs daily – and distributes tailored briefings to end users. The outage stemmed from corruption of the AIS-R database that feeds the rest of the system.

What caused the NOTAM system failure?

The FAA traced the root cause of the outage to contract personnel who were working to correct synchronization issues between the live NOTAM database and a backup database. Around 1 a.m. EST on January 11, technicians attempted to audit and repair database files, but this activity resulted in damage to the live database instead.

Specifically, a corrupted database file prevented new or updated NOTAM information from being disseminated to pilot briefings. The backups also contained the same corrupted file. So while NOTAM data was still flowing, it was incomplete and outdated. Once they identified the damaged file, the FAA was able to restore the database from an unaffected backup copy.

While early reports suggested a fire or cyberattack, investigators confirmed the outage was caused solely by human error – contractors executing improper database procedures. There was no evidence of sabotage or external intrusion.

What was the impact?

The NOTAM system failure led to major disruptions in domestic air travel on the morning of January 11:

  • Nearly 5,500 delayed flights
  • Around 1,200 canceled flights
  • Hundreds of thousands of passengers affected
  • Ripple effects continued throughout the day

The FAA ordered a nationwide ground stop from 5:28 to 9 a.m. EST, halting all domestic departures to allow time to validate flight information. Many carriers proactively began delaying and canceling flights even before the ground stop. Operations slowly resumed after the corrupted database file was repaired and integrity of the system was verified.

Delays and cancellations rippled across the country throughout the day as airline schedules returned to normal. Effects were felt most heavily along the East Coast due to the early morning timing, but ended up spreading throughout the national airspace system.

Could this outage have been prevented?

While human errors can never be fully prevented, there are measures that potentially could have avoided this outage:

  • Enhanced training – Additional training on proper database procedures may have prevented the mistake by contractors.
  • System safeguards – Safeguards could have halted database changes once anomalies were detected.
  • Decentralized NOTAM system – Distributing NOTAM processing across multiple sites could have avoided a single point of failure.
  • Faster cutover – A faster switch to contingency plans may have reduced the 90-minute delay in halting flights.

The FAA was already in the process of upgrading many aspects of the NOTAM system. While beneficial, these long-term upgrades would not have prevented this specific human error. Tighter controls around database access and changes may have reduced the likelihood of this outage.

What improvements are being made?

In the aftermath of the outage, Secretary of Transportation Pete Buttigieg stated that the FAA will address key areas for improvement:

  • Implementing redundancies in the NOTAM system to reduce single points of failure.
  • Enhancing cybersecurity measures to guard against external threats.
  • Reviewing internal policies around software management.
  • Improving response time and coordination during outages.
  • Updating training to strengthen technical skills.

While NOTAMmodernization efforts were already underway prior to this event, the FAA is expediting key upgrades, including transitioning to a more secure, web-based dissemination system. A multibillion-dollar contract was awarded in 2020 to replace NOTAM’s text-based system with more modern, graphical data delivery.

Enhancements to outage response plans and training are also underway. Checklists and procedures will aim to minimize delays in halting flights during future disruptions. More robust training will be required for personnel that manage pivotal aviation systems and databases.

Could this happen again?

Despite enhancements and safeguards, experts state that a NOTAM outage could certainly happen again. Like any complex, software-driven system, unanticipated failures can still occur due to:

  • Human errors
  • System bugs
  • Cyberattacks
  • Damage from fire, natural disasters, or power outages

While risks cannot be fully eliminated, the FAA is taking steps that will hopefully prevent repeats of this specific issue:

  • Restricting direct access to the live NOTAM database to reduce potential for operator errors.
  • Adding validation controls when applying changes to the live database.
  • Further separating the live and backup databases to isolate failures.

Enhanced employee training and new system redundancy requirements will also help minimize future disruptions. However, aviation experts concur that occasional outages cannot be ruled out given the enormous scale and complexity involved in managing the National Airspace System.

Could a cyberattack cause a similar failure?

Aviation systems clearly remain vulnerable to potential cyberattacks aimed at disrupting air travel. While the January 11 event was unrelated to hacking, a cyberattack conceivably could trigger similar cascading failures.

Some ways hackers could potentially disrupt the NOTAM system and flight operations:

  • Inserting fake or misleading NOTAM data into pilot briefings
  • Compromising integrity of flight navigation databases
  • Jamming critical communication frequencies
  • Damaging or disabling key infrastructure through targeted malware

The NOTAM outage highlighted that many FAA systems rely on outdated software, insecure operating systems, and legacy mainframe technology. Modernizing these systems is crucial to harden defenses against cyberattacks. Other recommended cyber measures include:

  • Network segmentation to contain failures
  • Multi-factor authentication
  • Artificial intelligence threat detection
  • Regular penetration testing

While daunting, cyber threats can be reduced through continued IT modernization and layered security tactics. The potential for malicious actors to disrupt aviation systems will remain an ongoing concern.

Conclusion

The FAA NOTAM outage on January 11, 2023 stemmed from human error during database maintenance work. Corruption of a key NOTAM system file prevented updates to critical flight safety information. This required halting domestic departures for over 90 minutes, leading to thousands of delays and cancellations.

While concerning, officials report no evidence of cyberattack or other nefarious activity. Investigations point to contractor mistakes during an attempt to correct synchronization issues. NOTAM modernization efforts already underway will help reduce, but not eliminate, future outage risks. This event highlights the fragility of the complex, interconnected aviation system.

After any major failure, lessons can be gleaned. The FAA is responding with improvements to outage response protocols, increased training rigor, additional system redundancies, and accelerated modernization initiatives. However, occasional disruptions will remain a possibility given the scope and cyber risks involved.