Discover · Technology

The largest IT outage in history came from one file that was missing a single field.

Format
Explainer
Happened
19 July 2024
Read
5 min
Sourced from CISA

What happened

On July 19, 2024, the cybersecurity company CrowdStrike pushed a routine update to its Falcon security software, installed on millions of Windows computers worldwide. The update was defective. Within minutes, an estimated 8.5 million Windows machines crashed into the "blue screen of death" and would not restart on their own, grounding flights and taking hospitals, 911 systems, banks and retailers offline, in what is generally regarded as the largest IT outage in history.

How one file broke that many computers

The update file the security software loaded did not match the number of inputs its own code expected, one field short. That mismatch caused the program to read memory it should not have, which crashed the entire operating system, not just the security software. Because Falcon runs with deep access to Windows itself (that access is the whole point of the product) its crash took the computer down with it.

Why the fix was not just a fix

Reversing the update meant physically reaching each affected machine, on a network that was often down, on computers that would not boot far enough to receive a remote patch. IT teams spent days manually rebooting machines into safe mode one at a time, at some of the largest organisations in the world.

What this has to do with Mach 9

The gap between "the software was tested" and "the software fails safely when something unexpected happens" is the entire subject of a cybersecurity course, and the day the phone does not stop ringing for a Cybersecurity Analyst.

Cybersecurity AnalystCybersecurity & Cryptography: Technology Foundation
More on technology