← Feed
The problem and fix for the FAA outage:
Digging through AI gobbledygook to come up with a coherent description of the problem, here’s my understanding:
The SMART tool rollout and an update to Chicago FAA FTI network forced a reboot across 20 year old Cisco 600 series routers and legacy Time Division Multiplexing (TDM) copper switches. As those routers attempted to reestablish routing tables the surge of reconnection traffic from the smart AI system that tries to pull in 22 data feeds at once limited processors and memory buffers causing them to output corrupted radar telemetry and pilot comms traffic. Because Chicago is central East-West hub, that routing instability rapidly cascaded across 22 regional air route centers.
This affected pings to determine if the system was up and running and pilot to tower communications over the FTI network.
TR:
So update the antiquated equipment but also the “SMART” system has no (or insufficient) backoff strategy or network monitoring to properly handle and report the problem. Also why were 22 centers updated at once?
Also I sure hope those data feeds are scrutinized with proper validation and error handling.
Replies (1)
-
@teriradichel@infosec.exchange I’m certain that they gave testing and verification as much thoughts as they did to rolling out while the systems were live. Which, I’m going to suggest on the basis of no evidence, is “nowhere near enough”.
I work with fairly trivial systems and when we recently upgraded a 20-year old codebase and moved it into the cloud we were extremely careful about going live and cutting over. Even though the worst case scenario was a delay of a day or two to the delivery of bulky furniture. Or maybe accidentally scheduling the same delivery twice.
Whoever did this was nowhere near concerned enough about what could happen when they pushed this out. Glad I’ve no reason to fly into the US anytime soon, that’s all I’m saying.
Open ##5537523