One customer changing their settings caused a major internet blackout, Fastly says

[Nevilledog]

Link to tweet

https://www.bbc.co.uk/news/technology-57413224

A major internet blackout that hit many high-profile websites on Tuesday has been blamed on a software bug.

Fastly, the cloud-computing company responsible for the issues, said the bug had been triggered when one of its customers had changed their settings.

The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet.

Fastly apologised and said the problem should have been anticipated.

The outage, which lasted about an hour, hit some popular websites such as Amazon, Reddit, the Guardian and the New York Times.

*snip*

[marble falls]

1. It's not that only a few companies 'run' the internet. It's that all these companies work ...

... to create perfect, elegant systems.

I worked designing windows and there were two theories on how to waterproof a window.
1. Seal it and make it leak-proof.
2. Admit water will work its way in eventually, so design a way to get it back outside.

Leak proofing works for only so long and resealing a window can actually have a negative 'patch' effect on function and will leak again.

Preparing for the inevitable ALWAYS works best.

Software and hardware crash and corrupt.

[mathematic]

2. I thought you were praising one humanity's greatest feats of technological engineering

But it seems like you're comparing the internet to a window designed to be leak proof, which confused the hell out of me because the internet was and continues to be designed according to your second principle.

To use your analogy, there was a leak in a few websites for a short time causing water to get inside and bring down the websites. The leak was fixed and the water was pumped out and the websites were back up in a short time with no water damage.

[marble falls]

3. Oh but I am. The internet is great. The companies maintaining it are the weakness ...

... there's just a culture of people feeling invincible, and above making mistakes and have an ability to fix anything and on the fly.

My comment is about recognizing that problems are inevitable and to get in front of them.

I think adding more companies to the mix is also a bad idea.

[Yavin4]

4. When you wrote, "I worked designing windows..."

I thought that you were going to talk about the operating system.

[marble falls]

8. I thought some might. Perhaps this is a litmus for generation: windows or Windows? ...

I identify geezer - windows.

I figure you're forty or so. The floppy disk suggests it. I wonder what a zip drive identifies?

Still got one and ten yellow zip disks.

[tanyev]

5. I hope that was one company and not one person.

[fescuerescue]

6. Oh I'm quite certain that it was one person at one company. And probably one character on one line.

You can peer review changes until the cows come home.

But in the end, it's a guy with a keyboard and tpyos happen.

This sounds like it was an internet routing issue. For instance, the difference between typing "/16" when you were supposed to type "/17" is 32,000 customers.

[hunter]

7. Once upon a time I accidentally crashed an entire university network from user space.

It was just a silly typo. That's how bugs are found.

I think I was kept around as some kind of lab animal.

They needed fools to test their foolproof systems.

[Wounded Bear]

10. Anybody who thinks they've designed a foolproof system...

is underestimating the creativity of fools.

[Politicub]

9. "But I just wanted to change my wallpaper!"

This must have been the mother of all settings.

[spanone]

11. Good to know the internet is so failsafe.

[Tommy Carcetti]

12. "The Internet! WE'RE ALL GOING TO DIE!!!"