So what the hell happened to CenturyLink nationwide and to 911 service across WA,

[pnwmom]

Montana, and elsewhere? They still don't seem to be saying.

Last night, many people in WA were awakened with an emergency message telling us that the 911 system wasn't working, and if we had an emergency we should call another number. It turns out that people had had trouble getting through 911 since the afternoon, and they finally got around to telling us. Great.

From what I understand, most people have 911 service back but they're not promising that it will keep working, probably because they don't know for sure what caused the outage in the first place. Or if they do, they're not saying.

https://www.theverge.com/2018/12/28/18159110/centurylink-internet-911-outage-fcc-investigating

A nationwide CenturyLink outage has knocked out 911 voice calls in parts of the US and affected everything from Verizon mobile data to ATM withdrawals, lottery drawings, and hospital patient records. The downtime, which also impacts CenturyLink’s residential internet customers, began Thursday and has now stretched on for more than 24 hours.

FCC chairman Ajit Pai announced Friday that such a delay is “unacceptable” and said the commission is investigating the ongoing outage after reports of people getting busy signals when dialing 911. “When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling.” Local authorities have taken to social media to share other emergency numbers, and some are also encouraging the use of Text to 911 if voice calls are unavailable wherever someone might need help.

CenturyLink has provided few specifics on the problem. The company first acknowledged a service disruption yesterday morning, later saying that “CenturyLink engineers and technicians identified a network element that was impacting customer services and are addressing the issue in order to restore services.” On Thursday night, CenturyLink estimated that the situation would be resolved within four hours — but that didn’t pan out.

On Friday, the company offered several “we continue to make good progress” updates but little in the way of new details. Service has been restored in some areas, but customer responses on Twitter suggest that many are still waiting for things to come back online.

SNIP

[dixiegrrrrl]

1. I can understand why they don't have answers yet

Could be a DDS attack.....

I love that we are all internet linked together, but that has a downside. I know when we lose service out here in the boonies, half the time it is because the single buried cable is somehow cut by somebody digging around ditch alongside the highway
or
our frequently severe thunderstorms. ( Lightening, flooding, etc)

And yet, when the puter id off for more than a couple hours, I find myself thinking...What did the orange menace do now?

[Major Nikon]

2. I've been hearing about it for the last couple of days

It’s been affecting lots of telecommunications services nationwide.

[pnwmom]

4. Right. The CL outage has been nationwide, but 911 systems in only some states are affected.

Also, at least one hospital in CO wasn't able to access its patient records.

[Major Nikon]

12. Some of our networks are unaffected

Seems to be affecting multiple backbone connections nationwide, but others aren’t affected at all. Reroutes could be affecting other services even if the paths themselves are up. Only so much data can get through.

[dixiegrrrrl]

6. I just got caught up on the story

trying to understand how 911 calls can be out in various places across the entire country because of one internet provider????

For some reason, I had picture in my head that internet providers had their own geographical market area.

and why no backup to 911 service?

Puzzling. Hackers? Poor planning? FEC is investigating now, but ...Trump. Sigh.

[pnwmom]

9. There SHOULD be a backup to 911 service. I read somewhere that CL was faulted

in 2014 for not having such a backup, and they were supposed to have corrected that.

In WA we automatically (after a period of years) have to send the contract out for re-bid, and another provider will be in place at the end of 2019. But we're stuck with CL for now.

[Major Nikon]

13. Backups can mean different things

They might have redundancy leaving the building, but at some other point you have a single point of failure. It can cost a shitton of money to get true redundancy from end to end depending on where it’s going and what infrastructure exists.

[pnwmom]

19. Okay, thanks!

[Major Nikon]

11. We have lost redundancy in our networks

Our network has sufficient backup paths so we have had only relatively minor outages. Within a general area it could be worse. From what I’m hearing the outages have been mostly resolved, but they still don’t know what the cause is.

[pnwmom]

20. We were hearing last night that the whole state of WA was without 911.

That doesn't sound minor to me.

And the CL service outage doesn't look minor either.

Link to tweet

[Major Nikon]

22. My company has nothing to do with 911

It’s not just 911 that’s affected, that’s just the piece that’s making the news. Centurylink serves many other networks. Our piece of it has had relatively minor impacts, but still affected across the US.

[pnwmom]

27. Sorry, I didn't understand that "we" referred just to your company.

I thought you meant "we" to refer to the country. Big difference!

[marybourg]

29. The map makes it look as though AZ

was in the affected area, but my landline and internet connection have been fine.

[pnwmom]

31. You might not have known unless you tried to dial 911. My internet and land line

were also fine, but we got a call notifying us that the emergency system wasn't working, and we should not call 911 in case of an emergency.

[marybourg]

35. Oh, I see. Fortunately I didn't need 911,

but didn't get such a call or text alert, either, nor was there any buzz amoing my neighbors

[uppityperson]

21. They were a landline and dsl provider first.

We couldn't call 911 for nearly 2 days, instructed via Twitter to text (landline) then drive to the nearest fire station or emergency services if we needed emergency help. Long distance of any sort was also out. Internet was dead slow via dsl.

[Autumn]

3. I have Century Link and all day yesterday internet connections in CO were slow and some links

I couldn't get to load.

[pnwmom]

5. There was a hospital in CO that wasn't able to access patient records. n/t

[dixiegrrrrl]

8. I can see another problem...

Our Dr. offices now send prescriptions to pharmacies, via internet.

[janx]

16. There were problems with banks, ATMs, etc.

All kinds of businesses were affected by this problem. And still no real explanation from CL.

[PJMcK]

7. Skynet is testing its new powers

It became self-aware late in the night several nights ago and, like an infant, it is searching its environment and abilities.


(snark)

[Andy823]

10. I received the alert

It was for Chelan and part of Douglas county. I live in Okanogan county and I guess we are OK, I looked and it was not on the list of counties I found that lost their 911 service. This really needs to be fixed so it never happens again.

[cilla4progress]

30. Hey hi!

Chelan Co. neighbor here!

[janx]

14. I posted about this a few times in LBN.

Centurylink (and formerly Quest when I lived in CO) is infamous for bad customer service. I was with them for 11 years in CO and have been a customer for 13 years here. I have a couple of short outages a couple of times a year here. So that's the preface.

Yesterday morning I woke to no internet. All lights on my modem were on, but neither one of my two computers could connect. Apparently the internet connection had died in the wee hours of the morning. I called CL, put the phone on speaker, waited for around an hour and was disconnected before I could talk to someone in customer service. Tried a few more times--same thing happened.

Then my phone (Verizon) started giving me a fast busy when I tried to call back AND when I tried a personal phone call. Texts on the phone sometimes did not go through.

I used my phone to get online using my data allowance and found that the same thing was happening to CL customers all over the country. CL had not issued an explanation as to what was going on, simply apologizing for issues with their customer service line. They did not address this massive internet outage that was happening nationwide! I looked at a couple of sites that track outages and was amazed at how far reaching this was.

Tom Udall posted a message for New Mexicans to call the FCC about this problem and even went as far as to post a number to call.

I was without internet for about 12 hours, and when it did come on, I could not call out via Verizon at all. This turned out not to be a coincidence. After some scrounging around online today, I found that Verizon uses CL for some data transfer here in NM.

When explanations from CL did surface later yesterday and earlier today, they were composed of jargon that wouldn't make any sense to the average customer.

[NurseJackie]

15. Sabotage?

[Generic Brad]

23. I'm thinking seasonal incompetence

I suspect there were a series of bad decisions made by technicians and managers filling in while many employees were taking a winter vacation.

[dembotoz]

24. Rule of thumb.. don't install anything from mid November to years end

Telcos in the past had use it or loose it Vaca...the people who could fix stuff were either deer hunting or on a beach.

[NurseJackie]

25. That seems more likely.

[Major Nikon]

28. My guess is a software update gone bad

Sometimes software updates work great in the lab, but problems surface when implemented more widely.

[lpbk2713]

17. FCC chairman announced that such a delay is "unacceptable". Now that's scary.

Better watch out. He might even raise his voice or shake his finger.

[C_U_L8R]

18. Who you gonna call?

...when Gozer the Gozerian shows up to party?

[FreepFryer]

26. A crucial national security/civil defense vector goes offline, and corporate shills reassure us.

Heads should roll over this.

[womanofthehills]

32. It pretty much knocked out our little NM town

We have a two block downtown and the majority of the stores had no service so they were only accepting cash. Luckily, the bank had service most of the day, but I was unable to check my account online.

[Maru Kitteh]

33. I would like to thank our new prezident Vladimir for the extra hour or so of sleep

last night - thank you comrade, I didn't really need to watch the factually correct, Kremlin-unapproved fake news after my 13-hour shift at the Veterans Home. Nope.

[CabalPowered]

34. There are some technical details on slashdot

Sounded like a cascading equipment failure with hints of a sophisticated denial of service attack.

[janx]

36. Do you have a link?

I tried to find that and could not.

[CabalPowered]

37. Here

https://tech.slashdot.org/story/18/12/27/2226253/users-report-of-nationwide-centurylink-internet-outage#comments

Which I think is from a reddit. Good discussion here:

https://www.democraticunderground.com/10142232214#post17

[janx]

38. Thanks much.

[Achilleaze]

39. It sucks to be vulnerable

And to realize that KGOP republicans have done absolutely doodley squat to protect or defend America from russian ACTS OF WAR, an ugly reality that all 17 US Intelligence Agencies agree upon. The KGOP republicans have betrayed America, and left us VULNERABLE.