Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix
5 40Microsoft says last week's massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage's worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."
"The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive."
So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.
5 comments
So incompetence on top of incompetence? (Score: 5, Funny)
by gweihir ( 88907 ) on Monday November 16, 2026 @11:39AM (#66426986)
Makes sense, good old Microslop.
First, a 14% capacity reduction should never be enough to kill things, even with that Teams failover. They obviously went cheap on infrastructure.
Second, a cascade-failure is what you want to avoid at all cost and that means capacity, monitoring, robustness and tests. Apparently too expensive.
And third, don't they test their stuff under overload? Don't answer that, doing so would cost money.
All in all the usual mode of giving tech details to obscure that they screwed up massively. Again.
Re:So incompetence on top of incompetence? (Score: 5, Interesting)
by SoonerC ( 6423252 ) on Monday November 16, 2026 @01:58PM (#66427268)
And third, don't they test their stuff under overload?
This 6 part article [substack.com] from Axel Rietschin goes into great detail about the problems going at at Azure and why they are happening. It's a clusterfuck.
Multithreading and race conditions. (Score: 5, Funny)
by Fly Swatter ( 30498 ) on Monday November 16, 2026 @11:40AM (#66426988)
Rust can't fix that one.
AI? (Score: 5, Funny)
by zlives ( 2009072 ) on Monday November 16, 2026 @11:59AM (#66427020)
why didn't they just use AI to fix it? why have humans spend 15 hours? AI would have done it in one second. I am Bender, Kill All Humans.
reminds me of a power grid "black start" (Score: 5, Interesting)
by v1 ( 525388 ) on Monday November 16, 2026 @02:12PM (#66427316)
Those are a pain because the power grid isn't star topology, it's web. This makes it difficult to isolate a few individual nodes to start up first. And once you DO get an "island" working again, you have to slowly add on other isolated segments. If you add too much at once, the whole thing comes crashing down and you have to start all over again.
It sounds like MS's "automated recovery" was set up to fail over to healthy nodes, which overloaded them and took them down too, just like you see when a power grid fails.
Everything is near max capacity when something trips, causing load to fail over to another path, which then overloads it, causing the entire combined larger load to fail over to another segment and quickly trip it, and they go down like dominos. Basically what happens if you fix one gateway and bring it back up, it gets mobbed and immediately "falls over".