National CenturyLink outage continues into Friday

[Omaha Steve]

Source: Omaha World Herald

By Aaron Sanderford and Nancy Gaarder

The widespread Internet outage for CenturyLink customers continued into Friday as the company’s engineers raced to fix a network element that disrupted many of its telecommunications services nationally.

Verizon customers in New Mexico and Montana also were affected, the Associated Press reported.

CenturyLink’s Twitter account offered the latest update shortly around 9:45 a.m. Friday, saying the company continued to make progress but that work to restore services on which many businesses and residents rely is taking longer than expected.

By midafternoon, CenturyLink said it had repaired the ability to reach 911 emergency dispatchers by phone in each of its markets.

Read more: https://www.omaha.com/news/nebraska/national-centurylink-outage-continues-into-friday/article_cf335e84-40dc-5859-b9f2-8fcc0a695d9b.html

[PoindexterOglethorpe]

1. Good grief.

Good thing I left NM yesterday morning and won't be back until after the New Year.

[janx]

3. The weirdest part was that

it affected not only Centurylink internet connections, but also Verizon phone service. Verizon apparently relies on Centurylink for transmission here. So my internet was out and phone was intermittent. Then the internet came back and I could not call out on my phone.

Udall was actually advising people on twitter to contact the FCC. He provided a phone number.

Centurylink customer service was down all day yesterday. No way to get through to a person. That made their customers livid--nationwide.

[PoindexterOglethorpe]

6. Century Link is my landline and internet,

but I have At&T for cell phone. It has long seemed a good idea to me to have different providers for different services.

[MurrayDelph]

16. Same for me

except I have Verizon. I'm glad I went with satellite TV, or we would have killed somebody by now.

As it is, I'm sitting outside the municipal library (not own yet), making use of their internet connection.

[KY_EnviroGuy]

2. Very disturbing situation....

when "a network element" in one company's IP-based system can shut down a substantial portion of America's crucial communications networks, including 911 services.

Being an old-fashioned engineer, I've always been an advocate of maintaining our original POTS (plain old telephone system) wire lines and central offices. AT&T and others will slowly take it away if we are complacent and let them.

..........

[area51]

4. +1 n/t

[AllaN01Bear]

5. the california puc has allowed at and t and others to do just that. hem.

[localroger]

7. That ship done sailed

POTS lines will be abandoned as they become too expensive to repair for whatever reason. AT&T will keep them going where it's cheap but they're not going to spend any money maintaining, much less extending, the infrastructure when radio towers are so much cheaper.

[PoindexterOglethorpe]

8. I don't get the people who proudly don't have a landline.

If cell towers go out, say from a hurricane, the landline will still work. Other things will take out the landline but the cell will still work. I hope to have both the rest of my life.

[reACTIONary]

10. POTS isn't the same thing as a landline....

..... POTS is twisted pairs strung overhead. You can get a land line that goes over, say fiber optic , like verizon fios. That's a land line, but it's not POTS.

And your land line eventually makes its way to a microwave tower, so even that depends on radio transmission.

[Nac Mac Feegle]

11. UMMM.... no

POTS is Plain Old Telephone Service.

Dial Tone line from a Switch. It can be carried over various types of facilities to reach the customer: Twisted Pair, Fiber to the Neighborhood (muxed), or even over a radio system.

Relatively little goes over microwave radio anymore. Usually special case type systems where expense and terrain issues come into play. The backbone is predominantly fiber.

Trust me, I've been doing this for more than 40 years.

I've seen things you cannot believe.

[KY_EnviroGuy]

13. Nac, my former business partner worked for the Bell System...

for a while back in the 60s as an engineer. He told me about how the central offices were built and how the battery systems were designed and maintained. Sounded as if an A-bomb hit them, they would survive so we could make phone calls.

Many of us older engineers still hold the original Bell System as one of our primary standards for electronics design reliability, but we're a dying breed.

[reACTIONary]

14. I would call what you are describing . ...

.... PSTN, publicly switched telephone network. Which might include cell phone, so it might not be the right term either.

POTS seems to imply the use of analog signaling over a copper loop. That's what I got from the wiki article. https://en.wikipedia.org/wiki/Plain_old_telephone_service

Of course, the way an informal term like POTS is used doesn't always allow for an exact meaning.

I didn't know microwave is not that prevalent. The advantage over rough terrain is easy to see, but I assumed it would be preferred over cables more generally.

And I am sure you have seen things that I would not, and would hope not to have to, believe !!!

[Anon-C]

9. Don't municipal fire codes require fire alarm systems to have a POTS line?

[KY_EnviroGuy]

12. Good question, Anon-C. Don't know the answer.

I've been distant from the fire codes for many years. Insurance companies and emergency services have input on many of those requirements and the NFPA and municipal codes implement them.

Would make sense to have copper to the TELCO central office for fire and security alarms for reliability. The killer of reliability with stuff in homes and businesses that use local digital modems or wireless services is power backup (battery) units, which few people maintain properly. That, and a lot of very cheap Chinese-made electronics.

[dembotoz]

15. not so muich anymore

while i prefer that good old fashioned pots i have increasingly seen other types of service approved for installation
You do get tired of getting stabbed in the back by alarm providers...you push post and the alarm folks tell the customer to just put it on the internet.
i now check with the customer and tell them to find out what the alarm company wants and i will get them the service.
att now is making a push thru there iot folks to use wireless for alarms.....

standards? we don''t need no sticking standards

[Nac Mac Feegle]

17. Update from an internal source

From their internal info:

1) Things are supposedly pretty much back to normal.

2) It wasn't technicians working on equipment: There's a "Moratorium" in effect that service affecting equipment is not going to be worked on between mid-December to early January. This is a hard and fast rule to protect the major travel and commerce seasons of the year.

3) Rumor is that one particular vendors' equipment is responsible for the failure; all of that equipment went down.

4) There was a massive conference call that lasted for more than 24 hours, and included people from technicians to the C.E.O.

5) They don't have the exact cause yet, but the Investigation will be massive, and thorough. Heads will probably roll.

6) Since Centurylink bought out/merged/was taken over by Level 3, there have been multiple downsizing exercises, resulting in the loss of a lot of people in a lot of different areas, some critical. A lot of knowledge was shoved out the door.

7) The remaining people are spread really thinly.

8) The problem is most likely to have been deep in the "backbone" systems, probably the failure (or lack) of a redundant system.


Welcome to the Corporate World, where "Businessmen" who "Know What They're Doing" are in charge.

Don't worry, we'll take care of things for you(TM). <snark>



Just some "rumors" from someone 'inside', that is relaying second and third hand information that may or may not be accurate.

[xor]

18. Any hint on the vendor that might be responsible, or any details on what the issue was?

Back when I worked at CL (in monroe) there was a lot of Calix hardware. Although, there was a lot of other stuff too due to them always buying out other networks. Green networks. Blue networks. Blah networks. Maybe all service provider networks are cluster Fs like that. I know Charter's network was a mess too, but my view into it was much more limited as I only worked in the headends and hub sites for my metro area.

[PoindexterOglethorpe]

19. I was an operator for Ma Bell, back when Ma Bell was around.

I know a bit about how the landlines worked back then. How calls were routed, how the mechanics of connecting calls happened. While I like a lot about the new technology, there's also a lot about the old technologies to appreciate.