On January 11, 2023, a major computer outage at the Federal Aviation Administration (FAA) caused a nationwide ground stop of all U.S. departing flights for several hours. This resulted in the cancellation or delay of over 11,000 flights, impacting millions of travelers. The FAA ordered all U.S. flights to delay departures until 9 a.m. EST as technicians worked to resolve the issue. This was one of the largest system failures in recent history for the FAA, which manages the air traffic control system across the country. The outage raised serious questions about the resiliency and reliability of the technology infrastructure that underpins the national airspace system.
What was the impact of the FAA computer outage?
The FAA computer glitch had wide-ranging impacts across the U.S. aviation system:
– Over 11,000 flights within, into, or out of the U.S. were delayed or cancelled. This represented approximately 1/4 of the 46,000 flights originally scheduled that day.
– More than 1 million air travelers faced disruptions, including missed connections and delays reaching destinations.
– Many travelers were stranded overnight when flights were cancelled. Airlines scrambled to find hotel accommodations for displaced passengers.
– Airlines estimated lost revenues in the tens of millions of dollars due to cancelled flights and lost bookings.
– With planes and crews out of position after cancellations, it took 1-2 days for operations to return to normal. The disruption rippled through the entire air travel system.
– Airline customer service centers were overwhelmed with calls as passengers tried to rebook cancelled flights. Wait times of several hours were common.
– Airports became extremely congested as outbound passengers were unable to depart. Security screening lines grew long.
What exactly was the problem with the FAA computer system?
The FAA outage was caused by damage to the agency’s Notice to Air Missions (NOTAM) system. NOTAMs provide critical safety information to pilots and airlines, advising them of closed runways, equipment outages, and other issues that could impact flight operations. Specifically, the FAA identified the outage as a “damaged database file” in the NOTAM system.
This critical database failed between 7-9 p.m. EST on January 10, preventing new or updated NOTAMs from being processed and disseminated. With outdated NOTAM data, the FAA could not ensure safe flight operations, requiring the ground stop. The damaged file corrupted both the primary and backup NOTAM databases simultaneously.
The FAA stated that there was no evidence of a cyberattack, sabotage, or other nefarious activity involved. The outage was traced to a single damaged database file, which investigators believe was caused by an internal technical glitch vs. external factors. Still, the simultaneous corruption of both the primary and backup databases raises questions about the FAA’s systems resilience.
How did the FAA resolve the NOTAM system outage?
FAA technicians undertook a number of steps to restore the NOTAM system:
– Reset the main NOTAM database and servers at the NOTAM processor facility in Auburn, Alabama.
– Activated the backup NOTAM database at the disaster recovery site in El Segundo, California.
– Verified integrity of the reset databases before using them to update and disseminate NOTAMs.
– Began processing the backlog of several hundred new NOTAM reports that had built up during the outage.
– Gradually added new or revised NOTAM data to ensure system stability.
– Coordinated updated NOTAM data with airlines, airports, and other aviation stakeholders prior to resuming normal operations.
– Monitored overall NOTAM system performance closely after resuming service to confirm stability.
By around 9 a.m. EST on January 11, the NOTAM system was declared restored and operational. However, unwinding the disruption across the national airspace took most of the day to complete.
What is the FAA doing to prevent another outage?
In the aftermath of the NOTAM failure, the FAA is undertaking a number of actions to improve reliability and prevent outages, including:
– Conducting a thorough failure review to determine the precise technical root cause. Detailed hardware and software checks will identify deficiencies.
– Evaluating the resilience of the primary and backup NOTAM databases, and adding redundancy where needed. Separating the databases geographically should prevent simultaneous corruptions.
– Enhancing NOTAM system monitoring tools to provide early warning of degradations. This could enable preemptive fixes before failures occur.
– Considering the use of advanced analytics techniques, like artificial intelligence, to spot anomalies in massive NOTAM data flows.
– Upgrading NOTAM database hardware/software infrastructure where aging technology could increase outage risks.
– Strengthening organizational focus on NOTAM operations, with clear accountability for uptime and emergency response.
– Auditing contingency procedures for critical systems across the FAA to ensure effective plans are in place. Practice drills will also improve response readiness.
– Exploring emerging technologies, like blockchain, that can securely disseminate NOTAM data directly to aviation stakeholders for greater decentralization.
– Partnering with airlines and industry groups to gather feedback on operational challenges from the outage and ways to enhance future collaboration.
These efforts will reduce the likelihood of another significant NOTAM failure. But realistically, the complexity of the air traffic control system means some risk always remains.
How old is the FAA’s air traffic control technology?
Many of the FAA’s air traffic management systems rely on legacy hardware and software that is decades old:
| System | Age |
| NOTAM Processor | Around 30 years old |
| Host Computer System | Over 40 years old |
| Display System Replacement | Over 20 years old |
While aspects have been upgraded, the underlying platforms trace back to the mainframe era. The Host Computer System, which routes flight plan data, still runs on a proprietary IBM mainframe database called Transaction Processing Facility (TPF). Replacing these aged systems is hugely expensive and risks operational disruptions during complex transitions.
The FAA has undertaken long-running modernization programs, like NextGen. But funding constraints and evolving technological requirements have slowed progress. Outages like the January 2023 NOTAM failure highlight the urgent need to upgrade obsolete infrastructure that cannot meet today’s demands for reliability and resilience. While a wholesale technology refresh would cost billions, the risks and impacts of failure for critical systems argue for major reinvestments.
Should the FAA’s technology systems be privatized?
Some industry observers have suggested privatizing FAA systems as one way to address the technology shortcomings exposed by the outage. But there are arguments on both sides of this issue:
Potential benefits of privatizing FAA technology infrastructure:
– Could improve access to technical talent by offering private sector salaries to attract qualified staff.
– May enable faster upgrades by avoiding cumbersome federal procurement processes.
– Private operators likely to be more disciplined about technology lifecycle management to avoid obsolescence.
– Reduce reliance on unpredictable federal funding cycles for major upgrades.
– Increased competition and alternatives to current FAA systems could enhance innovation.
Risks and downsides of privatizing FAA technology infrastructure:
– Replaces a public service mission with potential profit incentives if privatized systems must generate shareholder returns.
– Could reduce congressional and public oversight of critical national infrastructure.
– Privatization transitions often fail to deliver expected benefits in government settings.
– No guarantee private operators would invest more strategically in technology upgrades.
– Fragmentation across private systems risks complications for integrated air traffic management.
– Government maintains responsibility for aviation safety oversight regardless of privatization.
On balance, keeping core air traffic technology under FAA control seems the safer path forward. But major reforms in management, workforce development, procurement and O&M budgets are still required to evolve these critical systems for the digital era at the needed pace.
How can the FAA better prevent future outages?
Beyond immediate actions to address the NOTAM system, the FAA should undertake a holistic review of the technology infrastructure underpinning air traffic control to ensure reliability and resilience. Recommendations include:
– Conducting failure mode analyses of the highest-risk systems to identify and address single points of failure. Build sufficient redundancy into all systems essential for safe operations.
– Developing playbooks for different outage scenarios that codify steps for rapid response and recovery. Run simulated outages for practice.
– Creating stronger central governance of critical infrastructure changes to minimize uncoordinated modifications that increase risks.
– Investing in state-of-the-art monitoring tools that provide end-to-end visibility of systems operations and performance. Monitor legacy systems more aggressively.
– Building partnerships with private sector tech firms to tap innovations in cloud, AI and other emerging technologies applicable to air traffic management challenges. But maintain government oversight of essential functions.
– Working closely with global civil aviation authorities to share best practices for outage prevention and ensure coordinated contingency response plans. Outages do not stop at national borders.
– Exploring a bigger role for blockchain solutions to augment or replace centralized databases supporting essential services like NOTAM distribution.
– Assigning dedicated technical resources to focus on outage prevention and response across all critical FAA infrastructure. Make this a clear organizational priority.
With mounting flight volumes and complexity, past reliance on manual workarounds during outages may no longer suffice. Aging systems require robust modernization and automation for the FAA to fulfill its mission of safe skies for all.
Conclusion
The January 2023 FAA computer outage provides important lessons for strengthening the technology foundations of the nation’s air traffic control system. While the root causes are still under investigation, it appears to stem from aged infrastructure in need of upgrades. Beyond fixes to the specific NOTAM system, the FAA must prioritize modernization across its extensive, interconnected systems to prevent future failures. While risks cannot be eliminated entirely in such a complex enterprise, passengers expect air travel to remain safe and reliable during inevitable technology glitches. With proper resources and management focus, the FAA can retool its systems and operations to meet these expectations despite growing strains. But the status quo of underinvestment and piecemeal upgrades is no longer sustainable without operational impacts. The recent outage serves as an urgent wake-up call for decisive action to harden critical infrastructure supporting the world’s largest aviation market.