At 9:37 in the morning, Pacific time, on 28 February 2017, an Amazon engineer typed a command. It was a routine one, taken from an established playbook. The S3 billing system was running slowly, and the plan was to take a small number of servers out of service.

One of the inputs was wrong.

Instead of a handful of machines, the command removed a much larger set of servers. Among them was capacity for two subsystems that almost nobody outside Amazon had heard of. One kept track of where every object stored in S3’s Northern Virginia region actually lived. The other decided where new objects should go. Without the first, S3 in that region couldn’t answer the simplest question a storage service gets asked: where is this file?

For roughly four hours, a large part of the web behaved strangely. Images failed to load, apps stopped syncing, and services that had no obvious connection to Amazon suddenly stopped working. And the page meant to tell the world what was wrong couldn’t do its job either. According to Amazon’s own post-incident summary, the administration console of the AWS Service Health Dashboard depended on S3. For the first two hours, the company couldn’t update the status of individual services on its own status page.

The internet is often described as resilient, distributed and redundant, and it largely is. But again and again, the biggest outages haven’t come from earthquakes, cut cables or sophisticated attacks. They have come from a typo, a pattern of nine characters, a configuration file that grew too large, or a clock that ran backwards for a second.

This is a tour of those moments, and of what they reveal about the machinery we use every day without seeing it

A typo with a blast radius

The striking thing about the S3 incident isn’t the mistake. Engineers mistype things all the time. What stands out is what happened afterwards.

Once the servers were gone, the index and placement subsystems had to be fully restarted. Amazon’s summary explains that it hadn’t done a complete restart of these subsystems in its larger regions for many years. S3 had grown enormously in the meantime, and the safety checks that validate the integrity of all that metadata took much longer than expected. The index subsystem only had enough capacity to serve read and delete requests again at 12:26 p.m. The placement subsystem finished recovering at 1:54 p.m.

In other words, the system was designed to survive the loss of individual machines, and it did so every day without anyone noticing. It was much less practised at surviving the loss of a large share of itself at once, followed by a cold start.

Amazon’s fixes are as revealing as the cause. The tool was changed so that capacity is removed more slowly. A new safeguard prevents any subsystem from being taken below its minimum required capacity. And work on dividing the index subsystem into smaller independent cells, already planned for later that year, was moved up to start immediately. The status page’s administration console was rebuilt to run across several regions.

None of these changes makes typos less likely. All of them reduce how far a single typo can reach. Engineers have a name for that distance: the blast radius.

Nine characters that ate every processor

On 2 July 2019, at 13:42 UTC, an engineer at Cloudflare deployed a new rule to the company’s web application firewall, the system that inspects incoming requests for signs of attack. The rule contained a regular expression, a compact pattern used to search text.

Three minutes later the first alert fired. Processors handling HTTP and HTTPS traffic across Cloudflare’s entire network shot to nearly 100 percent. One of the company’s leaders reported that Cloudflare had lost 80 percent of its traffic. Machines that still had spare capacity returned “502 Bad Gateway” errors, because the processes that actually serve web traffic had stopped answering.

The culprit, as Cloudflare explained in a detailed post-mortem, came down to one fragment of a much longer expression:

.*(?:.*=.*)

Read loosely, it means “anything, followed by anything, an equals sign, and anything”. It looks harmless. The problem lies in how the firewall’s regex engine searched for a match. It used backtracking: it tries one way to split the text between the pieces of the pattern, and when that fails, it goes back and tries another. With several “match anything” pieces in a row, the number of possible splits explodes as the text gets longer.

Cloudflare’s own figures make the growth concrete. Matching the short string x=x against the simplified pattern took 23 steps. Matching x= followed by twenty x characters took 555 steps. A variant of the pattern ending in a semicolon took 5,353 steps for the same input. Real requests are much longer than twenty characters, and every request passed through the rule.

Two other details turned a bad regex into a global outage. The rule was meant to be deployed in “simulate” mode, which logs matches without blocking anything, but the expression still ran against live traffic. And firewall rules weren’t subject to the gradual, staged rollout Cloudflare used for its software, precisely because the firewall must react quickly to new attacks. The company’s configuration distribution system pushed the change to every location within seconds.

Recovery brought its own surprise. Cloudflare’s internal tools sat behind Cloudflare’s own access-control product, which was also affected. Some employees’ credentials had been disabled by security timeouts because they hadn’t used them recently. Engineers had to fall back on a rarely used bypass procedure that, in the post-mortem’s words, wasn’t well-trained on.

At 14:07, the team switched off the web application firewall globally. Two minutes later, traffic and CPU were back to normal. The outage had lasted 27 minutes.

The bug that waited four weeks

Some mistakes go off immediately. Others wait.

On 12 May 2021, the CDN provider Fastly deployed a software update. It contained a bug that nobody noticed, because it only appeared under a very specific combination of circumstances. For almost four weeks, nothing happened.

Then, early on 8 June, a customer pushed a configuration change. Fastly was careful to describe it as valid: the customer did nothing wrong. But it happened to contain exactly the conditions needed to trigger the dormant bug. At 09:47 UTC, 85 percent of Fastly’s network started returning errors.

For an hour, many well-known websites, including major news outlets and government sites, displayed bare error pages. For many readers this was the first time they learned that sites they thought of as independent all passed through the same company’s servers.

Fastly’s response was quick. The disruption was detected within one minute, and the customer configuration was identified at 10:27. Within 49 minutes, 95 percent of the network was operating normally. The permanent bug fix began rolling out that same afternoon.

The lesson is subtle. The trigger wasn’t the mistake, and the mistake wasn’t the trigger. A latent bug and a perfectly ordinary action had to meet. Testing can’t easily anticipate every combination that thousands of customers’ configurations might produce.

The second that ran backwards

Computers are good at counting time, but terrible at handling time that doesn’t behave.

Every so often, the world’s timekeepers add a leap second to keep atomic time aligned with the Earth’s slightly irregular rotation. The clock reads 23:59:59, then 23:59:60, then midnight. Many systems can’t represent a minute with 61 seconds, so their clocks deal with it by repeating a second or stepping backwards.

On 1 January 2017, at midnight UTC, that is exactly what bit Cloudflare’s DNS service. Its in-house DNS software, written in Go, measured how quickly upstream servers responded by subtracting one timestamp from another. When the clock stepped back by a second, a lookup that started just before the leap second and finished just after it appeared to have taken a negative amount of time.

That negative number fed into a smoothed performance estimate, which itself eventually went negative. The value was then passed to a random-number function, rand.Int63n(), which only accepts positive numbers, and the program panicked. The impact was limited, but real: at the peak, about 0.2 percent of DNS queries to Cloudflare were affected, mainly for customers using CNAME records. The incident lasted until 06:45 UTC.

The underlying issue was well known among Go developers. At the time, the language’s standard time function didn’t guarantee a monotonic clock, one that only moves forward. Go 1.9, released later in 2017, changed that: time readings now include a monotonic component, and subtracting two of them measures elapsed time correctly even when the wall clock jumps.

The leap second itself is also on its way out. In 2022, the General Conference on Weights and Measures voted to stop adding leap seconds by 2035, in large part because of the trouble they cause computer systems.

How a country made YouTube disappear

Every incident so far happened inside one company’s infrastructure. The next one shows how the internet’s routing system can spread one network’s decision to everyone else.

The internet is a network of networks. They find routes to each other through the Border Gateway Protocol, BGP. Each network announces which blocks of IP addresses it can deliver traffic to, and its neighbours pass those announcements on. Routers follow one simple rule when announcements overlap: the more specific one wins. A route to a small block of addresses beats a route to a larger block that contains it.

On Sunday 24 February 2008, Pakistan Telecom was ordered to block YouTube inside Pakistan. At 18:47 UTC, it began announcing a route for 208.65.153.0/24, a block of 256 addresses that was part of the larger 208.65.152.0/22 block YouTube itself announced. The announcement was meant to steer local traffic into a dead end. But it leaked: the Hong Kong carrier PCCW Global passed it on to the rest of the world.

Because the hijacked route was more specific than YouTube’s own, routers around the world preferred it. Traffic for YouTube was sent towards Pakistan, where it disappeared.

The timeline reconstructed by the RIPE NCC, the regional internet registry for Europe and the Middle East, reads like a counter-attack in slow motion. At 20:07, YouTube began announcing the same /24 block to compete on equal terms. At 20:18, it went further, announcing two even more specific /25 blocks, which won under the longest-prefix rule. At 21:01, PCCW withdrew the routes originating from Pakistan Telecom, and the hijack ended.

The episode became a textbook case for why routes need validation. Systems such as RPKI, which lets address holders publish cryptographically signed statements about which networks may announce their addresses, are designed to make this kind of leak easier to reject.

The company that locked itself out

On 4 October 2021, Facebook, Instagram and WhatsApp vanished for roughly six hours. In this case nobody else hijacked anything. Facebook effectively withdrew itself from the internet.

According to the company’s engineering post, an engineer issued a command during routine maintenance to assess the availability of global backbone capacity. Instead, it disconnected all the connections in Facebook’s backbone network, the private network linking its data centres. A tool designed to audit commands like this and stop dangerous ones had a bug, and didn’t stop it.

What followed was an automatic and logical reaction. Facebook’s DNS servers are designed to withdraw their BGP announcements when they can’t reach the data centres, on the reasonable assumption that an unhealthy location shouldn’t receive traffic. With the backbone down, every one of them concluded that it was unhealthy. They all withdrew their routes, and the rest of the internet could no longer reach Facebook’s DNS servers, even though the servers themselves were running.

Without DNS, the names of Facebook’s services led nowhere. The same failure also broke many of the internal tools engineers would normally have used to diagnose and fix the problem. Both normal and out-of-band network access were down, so engineers had to travel to the data centres. The company noted that these facilities are deliberately hard to get into, and that their hardware is designed to be difficult to modify. That is excellent day-to-day security, and a slower recovery on the one day you need to get in quickly.

There was one final, physical twist. Bringing everything back at once was risky. With services down, the data centres had seen their power consumption drop by tens of megawatts, and suddenly reversing that could have endangered everything from electrical systems to caches. Services were brought back gradually, following procedures the company had practised in “storm” drills.

A file that doubled overnight

The most recent large example shows the same pattern, adapted to modern infrastructure.

On 18 November 2025, at 11:05 UTC, Cloudflare changed the access permissions of a ClickHouse database cluster. The change was intended to improve security. As a side effect, a query that generated a configuration file for Cloudflare’s bot-management system began returning duplicate rows, and the file roughly doubled in size.

The proxy software that handles customer traffic had a limit of 200 features for that file, comfortably above the roughly 60 it normally contained. The doubled file exceeded the limit. When the new proxy loaded it, the bot-management module panicked, and requests started failing with HTTP 5xx errors.

What made the incident confusing was that it flickered. The file was regenerated every five minutes, and only the parts of the database cluster that had already received the permission change produced the bad version. For a while, Cloudflare’s network alternated between good and bad files, recovering and failing again. That pattern initially looked more like an attack than a configuration problem. Cloudflare’s post-mortem stated plainly that the outage was not caused by any cyber attack or malicious activity.

Core traffic was largely restored at 14:30, when a known-good file was deployed globally. Every downstream system was back to normal at 17:06.

The pattern behind the accidents

Put the incidents side by side and they stop looking like isolated bad luck.

<table> <thead> <tr> <th>Incident</th> <th>What actually went wrong</th> <th>Why it spread so far</th> </tr> </thead> <tbody> <tr> <td>Amazon S3, February 2017</td> <td>An incorrect input to a server-removal command</td> <td>Core subsystems lost too much capacity at once and needed a slow cold restart</td> </tr> <tr> <td>Cloudflare, July 2019</td> <td>A regular expression with catastrophic backtracking</td> <td>Firewall rules were pushed globally within seconds, without staged rollout</td> </tr> <tr> <td>Fastly, June 2021</td> <td>A dormant software bug, triggered by a valid customer configuration</td> <td>The same software ran across the whole network</td> </tr> <tr> <td>Cloudflare DNS, January 2017</td> <td>A leap second made measured time negative</td> <td>Code assumed time could only move forward</td> </tr> <tr> <td>YouTube, February 2008</td> <td>A route intended for local blocking leaked to the world</td> <td>BGP prefers more specific routes, and neighbours passed it on</td> </tr> <tr> <td>Facebook, October 2021</td> <td>A maintenance command disconnected the backbone, and an audit tool failed to stop it</td> <td>Automatic health logic withdrew every DNS route, and recovery tools depended on the same network</td> </tr> <tr> <td>Cloudflare, November 2025</td> <td>A permissions change doubled a configuration file beyond a hard limit</td> <td>The file was regenerated and distributed globally every few minutes</td> </tr> </tbody> </table>

Three themes keep returning.

Speed is a double-edged sword. The systems that push changes worldwide in seconds are what let providers respond to attacks and fix problems quickly. They also let a bad change reach everywhere before anyone notices. That is why the post-mortems keep returning to staged rollouts: deploy to a small fraction of servers first, watch, then expand.

Safety systems can turn against you. Facebook’s DNS servers withdrew their routes because they were designed to protect users from unhealthy data centres. Cloudflare’s credential timeouts existed for security. Both were good ideas that, under unusual conditions, made recovery harder.

Recovery tools often depend on the thing that broke. Amazon’s status console needed S3. Cloudflare’s internal tools sat behind its own access product. Facebook’s diagnostic tools needed its own DNS. Every one of these dependencies was invisible until the day it mattered.

Why this matters to anyone running a server

These stories come from some of the largest infrastructure companies in the world, but their logic scales down surprisingly well. A small hosting provider can also have a status page hosted on the same server it monitors. Its backups can depend on the storage system they are supposed to protect. A configuration change can be applied to every tenant at once by a script that nobody reviewed. A firewall rule can be added at three in the morning to stop an attack, with nobody quite sure what it will do to normal traffic.

The improvements described in these post-mortems aren’t exotic. Change the system gradually, so a mistake hits a few machines rather than all of them. Put limits on what a single command can do. Keep the tools you need for recovery outside the system that might fail. And practise starting things from cold, because some systems only fail completely once a decade, and by then nobody remembers how to bring them back.

Somewhere right now, an engineer is about to press Enter on a perfectly routine command, with one input slightly wrong. The internet will survive it. Whether a few servers notice or half the web does depends on decisions made long before that keystroke.