FAA Router In Middle-of-nowhere Arkansas Identified as Cause of Nationwide Flight Chaos
11 100Though the nationwide flight-grounding ended Friday, "Nearly 2,300 flights in the U.S. have been cancelled Sunday in the aftermath..." reports ABC News. "Additionally, some 18,689 flights were delayed, per FlightAware."
But "a single malfunctioning FAA router in rural Arkansas appears to have caused the air travel chaos," reports the New York Post: The problem originated at an FAA radio site in El Dorado, Arkansas, where a newly installed router flooded the network with excessive data traffic, disrupting communications between pilots and air traffic controllers, the Wall Street Journal reported. "We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds," Rebecca Guy, the FAA's technology chief, said on Saturday. .. [U.S. Transportation Secretary Sean] Duffy said the failure occurred during routine maintenance to replace aging copper telecommunications with fiber-optic cables — part of a broader effort to modernize the FAA's outdated infrastructure. The router began flooding other devices across the air traffic control network with connection attempts, disrupting both communications with aircraft and radar feeds. A radar station in Amarillo, in the Texas panhandle, is also being eyed as contributing to the outage.
And new flight cancellations reported for Monday already stood at 144 as of Sunday afternoon, with 641 delayed, reports ABC News (citing data from Flight Aware): A spokesperson for Hewlett Packard Enterprise, which owns Juniper, the company that manufactured the routers believed to be central to the ground stop, said in a statement to ABC News that the devices were working properly Friday, when the ground stop occurred. "Based on what we know today, the device functioned as it was configured to by the installer. It did not fail or act improperly," the statement said...
L3 Harris is a government contractor that provides technology and IT support for the Federal Aviation Administration (FAA) and that worked with Hewlett Packard Enterprise and Juniper to install the routers. "We have that router in the L3 Harris lab down in Florida," Transportation Secretary Sean Duffy told co-anchor Martha Raddatz on ABC's "This Week" Sunday, referring to the specific router in Arkansas that officials believe caused instability in the air traffic communication system Friday afternoon.
"L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously," L3Harris said in a statement Saturday.
11 comments
What is this bullshit? (Score: 5, Insightful)
by MachineShedFred ( 621896 ) on Sunday November 15, 2026 @08:15PM (#66426038)
If it only takes a couple thousand connections a second to shitcan your entire network, your network is garbage.
This "explanation" makes absolutely no sense. SMB routers easily purchaseable at Best Buy can do better than this. This sounds like they fucked something up and just don't want to admit it.
Re:What is this bullshit? (Score: 5, Insightful)
by rahmrh ( 939610 ) on Sunday November 15, 2026 @08:43PM (#66426066)
Odds are they looped the network (I have seen it happen on other wan router upgrades/wan wiring upgrades) it cause severe network havoc and the connections pass thousands fast.. I have also see the network issue cause disconnects and reconnects on the backends and each of those new connections create new processes and use ram and it only takes that going on for a while to crush the backend's cpu and ram (because the old processes do not go away fast enough).
Re:What is this bullshit? (Score: 5, Interesting)
by TWX ( 665546 ) on Monday November 16, 2026 @10:02AM (#66426860)
Odds are they looped the network (I have seen it happen on other wan router upgrades/wan wiring upgrades) it cause severe network havoc and the connections pass thousands fast.. I have also see the network issue cause disconnects and reconnects on the backends and each of those new connections create new processes and use ram and it only takes that going on for a while to crush the backend's cpu and ram (because the old processes do not go away fast enough).
If they managed to loop a network that uses a routing protocol then I'd be very impressed. They are using L3 links, right?
I've dealt with L2 networks that had loops leading to broadcast storms, but the last time I saw a network that used multi-hop L2 across WAN links was a legacy topology that derived from the prior Novell Netware network, with one TCP/IP router across the whole organization for the enterprise zone. It frankly was a shiatshow because even though each site had its own VLAN, the L3 router could end up swamped if there was a loop at any of the nearly 100 sites. For reasons I still don't understand the prior network engineer had disabled spanning-tree to the access port level at the sites as well.
In a L3 network if the routes are configured statically then I could see it being possible to introduce some kind of routing loop, but we've had reliable routing protocols for decades now. We can even construct the network with an overlay/underlay model using older technologies like MPLS or newer ones like EVPN if one wants to get fancy, allowing for the underlay to handle the convergence and other control-plane messaging with the data-plane traffic nicely segmented on the overlay. Juniper supports this, and if there's a mix of HPE and Juniper gear in this network, the HPE stuff should still support it.
So that this happened on a Friday afternoon is unconscionable. This is stuff that should be reserved for a minimal-traffic period if an actual downtime window isn't possible, when as many of the redundant systems (ie older physical radios for controllers to use) are up and on standby. That this happened during one of the busiest times of the week is ridiculous, and the timing alone calls into doubt the claim of what happened.
Re:What is this bullshit? (Score: 5, Insightful)
by high_rolla ( 1068540 ) on Sunday November 15, 2026 @08:46PM (#66426070)
Indeed. It does seem like something bigger is at play here and they're scrambling to try and hide it.
But if you're going to make up a dummy excuse for something of this magnitude and publicity you really do need to make sure you come up with a fairly plausible one as it is fairly obvious that many technical people are going to evaluate any excuse you give. (however this may be difficult if you've fired all the competent staff)
I really hope the media keeps up with this and doesn't get distracted when Trump does his usual "create some new distraction to deflect from the current crisis".
Re:What is this bullshit? (Score: 5, Insightful)
by Berkyjay ( 1225604 ) on Sunday November 15, 2026 @09:27PM (#66426116)
I'm still convinced this is just a play by the Trump admin to get $50 billion from Congress that absolutely will never go towards ATC upgrades.
Re:What is this bullshit? (Score: 5, Informative)
by rta ( 559125 ) on Sunday November 15, 2026 @11:26PM (#66426258)
If it only takes a couple thousand connections a second to shitcan your entire network, your network is garbage.
This "explanation" makes absolutely no sense. SMB routers easily purchaseable at Best Buy can do better than this. This sounds like they fucked something up and just don't want to admit it.
they're not IP routers and we're not talking about TCP/IP sockets here. This is old T1 / T3 etc. connection oriented stuff from the 80s and 90s where the virtual circuit set-up and teardown were on a side channel. Thousands of such connections a second is significant.
Re:The router wasn't the problem, project 2025 was (Score: 5, Interesting)
by JBMcB ( 73720 ) on Sunday November 15, 2026 @08:50PM (#66426080)
The Trump administration as part of project 2025 fired virtually all of the competent people so they could replace them with people who are loyal to Trump above All Else.
This is the FTI network, originally built by Harris around 2002, and now maintained by L3Harris under contract to the FAA. The upgrade that caused the problem is part of a modernization project approved at the beginning of the year in response to the system crash that took out Newark last year.
The FAA has very little to do with the maintenance of FTI. It's all contracted out to L3Harris. L3Harris says they need an upgrade to the network, congress appropriates money for it, and the FAA hands it to them.
If anyone else was president, the exact same chain of events would have happened. L3Harris botched a router config. It happens all the time.
That's not to say the head of the FAA isn't incompetent. It's just that this isn't an example of it.
Re:The router wasn't the problem, project 2025 was (Score: 5, Informative)
by bill_mcgonigle ( 4333 ) * ) on Sunday November 15, 2026 @10:11PM (#66426176)
Did they not budget for a test network in a lab for the ATC system?
The first rule of government contracting is always charge double for building two.
We are not getting the full story.
Re:The router wasn't the problem, project 2025 was (Score: 5, Insightful)
by haruchai ( 17472 ) on Monday November 16, 2026 @01:03AM (#66426342)
they fired too many senior people and handed control to an AI system earlier this week.
Re:The router wasn't the problem, project 2025 was (Score: 5, Insightful)
by UnknowingFool ( 672806 ) on Monday November 16, 2026 @09:33AM (#66426786)
Some of us would argue that it might not have happened under any other president as this one insisted on indiscriminately removing as many people as possible. Some of those people might have prevented it. We might not know what exactly happened as this administration seems to avoid oversight and scrutiny.
Actual cause: PRD maintenance during the day (Score: 5, Insightful)
by Dictator For Life ( 8829 ) on Sunday November 15, 2026 @08:28PM (#66426058)
You never do maintenance on production during the day. Especially a Friday.