ONLY 56 technical ways twitter could fail.

[usonian]

Last edited Sun Nov 20, 2022, 12:57 PM - Edit history (1)

Seriously, Large websites are beasts that require complex architectural design and constant monitoring and maintenance. Details at the link below.


https://threadreaderapp.com/thread/1593541177965678592.html

As an SRE (Site Reliability Engineer) and sysadmin with 10+ years of industry experience, I wanted to write up a few scenarios that are real threats to the integrity of the bird site over the coming weeks.

For context, I have seen some variant of every one of these problems pose a serious threat to a billion-user application. I've even caused a couple of the more technical ones. I've been involved with triaging or fixing even more.

On edit:
Not meant to ignore the business/social failure modes.
Those will become more than obvious!
It’s just that most people don’t appreciate the complex stuff that’s generally out of sight.

[C_U_L8R]

1. Add: advertisers flee, cash flow stops, bank accounts empty, credit goes poof.

[usonian]

2. Yes, this is tech-focused stuff, for the people who might think it's simple to run.

A “runbook” is like an NFL playbook.

[Irish_Dem]

3. The technical people are missing the whole social psychology aspect of twitter.

Twitter is not just a coding platform, is a unique gathering of PEOPLE.
It is a SOCIAL platform.

The advertisers who pay to make Twitter viable are PEOPLE.

The coders who keep Twitter up and running are PEOPLE.

If Musk is firing everyone but the coders, the company is doomed.

[2naSalit]

5. +1

[drray23]

7. indeed. some of these scenarios are outlined in that threadapp

A lot of situations having to do with having the right people for moderation, policy decisions, dealing with foreign governments, various regulations, etc. etc.

[Irish_Dem]

8. I only saw the technical coding part, and gave up reading the rest.

It looked ridiculous to me.

Missing the entire point of Twitter.

[Hermit-The-Prog]

13. Musk is missing understanding all the things in that thread.

Either he's deliberately trying to destroy it, or he naively thinks it's a simple hardware + software machine that just needs a few people to change the oil now and then.

To clarify: The point of this thread is *not* that these problems are unique to Twitter. Not at all. Any sufficiently complex/popular technical system will face some/all of these scenarios, and smart people have set up great existing processes for addressing *all of them*.

Twitter already has those processes too. The point is: facing these problems with a skeleton crew *without sufficient time for a transfer of tribal knowledge* is a recipe for disaster. You need enough boots on the ground, and you *need* people who know the magic incantations.

[Irish_Dem]

15. Right we don't know if it is deliberate sabotage or total incompetence.

[Ford_Prefect]

19. Never ascribe to the complex what is better explained by the mundane and obvious in this situation.

Elon is greedy, vain, and so libertarian as to be himself the stereotype of the IT purist.

Of course he is incompetent to deal with the real needs of dynamically operating Twitter. He lacks the simple humility it takes to deal with people, ANY people. A platform which must support human expression, along with the complexities of culture and language is far outside his lane.

He sees Twitter as a process with rules that need to be set in stone with limited pathways of operation and use. He imagines he will ultimately "cure" Twitter of its faults. This is why he's fired, and now chased away most of the people who were keeping Twitter operational. He thinks they were the cause of its instability because they were constantly altering the platform in small ways which kept it from being the efficient cash cow he expects it to be.

Elon doesn't understand how people communicate. He believes that it can be stated in code once and for all and that result will be the platform of Twitter 4.0* from now until the next millennium. Any subvariants will be strictly according to the initial coding, and that will keep the situation in hand.

WOW........Just WOW. How massively wrong is that?


*(I don't actually know how many versions of Twitter exist as I don't inhabit the Twitterverse.)

[Irish_Dem]

23. Yes the Law of Parsimony: The simplest explanation is often the best.

[Ford_Prefect]

26. I neglected to state that my observation of Elon makes him out to be a religious fanatic. Oops.

Of course Parsimony would mean that he is, in fact, a religious purist when it comes to Twitter.

Imagine...

[Irish_Dem]

27. Yes he is some sort of pathological purist.

We just aren't sure which kind.

[chriscan64]

33. I don't think the point of the article was about the point of Twitter.

When we think of the overarching effects or purpose of Twitter, we are considering Twitter in its normal functioning state. This is about the nuts and bolts of Twitter's insides that keep it in that state. If the discussion was about a hypothetical Revolutionary War Era version of Twitter that used hand-written scrolls delivered on horseback, this article would have been about the mechanical things that could go wrong with that system. What if a horse breaks a leg? What if a scribe runs out of ink when the ink shoppe is closed, etc.

Yes, Twitter is not in total about the technology, but it is of technology. This article is about the things that could go wrong with the technology that it uses.

[Irish_Dem]

34. Most things being written are about Twitter technology.

I wish there would be more about the social psychology aspects of Twitter.

Like in your example, is everyone going to go on and on about the horse and wagon carrying the scrolls?
When are we going to talk about what the heck is the content of the hand written scrolls?

No one cares, we are going to talk about the horses endlessly.

[chriscan64]

35. I also would like to see more about the social psychology aspects.

My guess is that the preponderance of articles about the mechanics is due to the fact that the mechanics are currently at risk of failure. The risk is a current event, which will draw out the proliferation of articles.

Ultimately, the discussion should be about the content. The whole fiasco started from Musk's dissatisfaction with the content. If the only thing he did was to alter the content, it would be the only topic of debate. However, he also tinkered with the work force which has led to the risk of having no content to discuss.

[Lonestarblue]

10. Yes, it's the online version of the town square or the town hall.

[Irish_Dem]

12. Yes it is about people, not a coding company.

[orthoclad]

21. Correct, it's community,

not code. But the code is the DNA of the organism.

This is what happens under capitalism when the townsquare is owned for profit. Like the libertarian notion of private bridges and roads.

(and hiya)

[Irish_Dem]

24. I am arguing the opposite. The DNA is social. Coding is merely a tool to keep it going.

[orthoclad]

28. The complexity of the code

is very much like biochemistry, with a million interactions on the molecular level compensating for each other. The under-systems described in the thread compare to organs cooperating to keep a body going.

DNA is literally code: the code which tells the organs how to function. It's a physical code with physical expression.

I see the social/community aspect as the total organism.

These are metaphors: on one level, there is underlying physical complexity/underlying code, and on another level we have the whole organism: a living speaking person, or a living speaking community of people interacting. But if the pancreas or the database break, the person/community breaks.

He describes a lot of the social implications of the code breaking: users leaving because it gets too slow, criminal cases filed against twitter for evil content, suicides, and accidentally enabling genocide when the moderators don't have the software tools they need to oversee the content. Compare to diabetes or thyroid malfunction.

But to me, the main point is private ownership of a global public service/community. I saw someone say "NATIONALIZE TWITTER". I would consider participating if it isn't driven by profit. It's like having Amazon owning and operating town hall.

(edited)

[Irish_Dem]

32. Just different metaphors for the same thing.

My field is psychology. I see humans as social animals. They will always find a way to interact.
No matter what century they live in. No matter the technology.

Like transportation. People will always move from Point A to Point B.
But the methods to do so change with the times.

People drive cars today, or fly in airplanes.
But there is not an organic combination of humans with the mechanics of the car.

But I understand the points you are making with the biology metaphor.

[orthoclad]

37. Yes, a matter of viewpont

I'm running on the fact that actual DNA is literally code: it's wet software enbodied in molecules, not chips or disks.

I'm guessing that you're using the colloqial, metaphorical "DNA", referring to the actual truth of a thing, the ding an sich, could I say "backbone", whereas I'm struck by the similarity between actual deoxyribonucleic acid and software code.

I agree: the true nature of Twitter is communication and interaction; the code is just the corporate substrate. What we need to protect is the community -- not the corporation.

Maybe the UN could buy it and run it as a global post office, after Cheechon Chusk "sinks" it. Better than letting Bonesaudi Arabia, the second owner, take it.

(for reference -- Cheech and Chong were a stoner comedy team. I think Elmo is chronically stoned, it would explain some behavior. Appropriate for a comedy team, but for global complexity?)

[ancianita]

39. Cool. I take it you're into using code to secure users over corporate proprietary software?

I'm guessing that you're using the colloqial, metaphorical "DNA", referring to the actual truth of a thing, the ding an sich, could I say "backbone", whereas I'm struck by the similarity between actual deoxyribonucleic acid and software code.

I agree: the true nature of Twitter is communication and interaction; the code is just the corporate substrate. What we need to protect is the community -- not the corporation.

Could you say you're into using code to secure users over corporate proprietary software?

[ancianita]

40. Well, as a utility, nationalized for the public ...

I saw someone say "NATIONALIZE TWITTER". I would consider participating if it isn't driven by profit.

... couldn't a unionized coder/worker coop run such a thing and share the profits?

[orthoclad]

46. Good idea!

[paleotn]

25. Granted, but just the hardware and software....

could put Twitter in the proverbial ground right quick. Point is, the company is doomed by everything you mentioned, plus a bunch of stuff we don't even know about.

[Silent3]

43. I think you're missing the point of the article

There's nothing at all in this article that says anything for or against Twitter being a social platform.

This is a technical guy talking about the many technical ways a big, complex web application can fail.

The fact that Twitter is a social platform doesn't distance Twitter from technical problems, as if human warm fuzzies make the site more durable against outages and data loss and privacy breaches.

If you want to write about all of the social ways Twitter can fail (and has failed in the past), go for it. Write your own article. But the author here isn't guilty of some terrible oversight just because he chose to decided to write about his own area of expertise.

[Hermit-The-Prog]

4. No self-respecting BOFH is limited to 56 ways ...

http://bofharchive.com/BOFH.html

[TxGuitar]

6. 56 ways before breakfast maybe

If he sleeps late.

[Pluvious]

9. Wow, that was brutal

It’s just a matter of time…

Sad

[Obvious85]

11. there are 50 Ways To Leave Your Lover

As Paul Simon famously sang. this is a great article, maybe Paul needs to update his tune

[Bernardo de La Paz]

14. Very good read. . . . nt

[ancianita]

16. Wow. Thanks.

[republianmushroom]

17. Interesting

[NQAS]

18. That was fascinating reading.

Terrifying in many ways. But fascinating.

Answers the question, hoe hard can it really be to run Twitter?

Answer: really, really hard.

[krispos42]

20. I just explained to my mom why I thought Twitter would collapse by end of the year

Then I read this... It might not survive to December...

[patphil]

22. I was a System Administrator in a QA/QC organization for a Fortune 500 company.

Although I wasn't on the IS side, I knew many of them, and did a whole lot of validation testing for our laboratory information management systems.
I also was responsible for maintaining the actual data and users at the lab level.

I can tell you, this guy is telling it like it is. Imagine several of these scenarios happening in a cascade style event. It could be bad enough to permanently end Twitter.
The legal liability worldwide could be in the multi-billion dollar category, and tie the company up in courts for decades.
This is what can happen when a totally naive idiot makes snap decisions for a company he doesn't understand, and doesn't care to.
He's essentially an entrepreneur, not possessing the skill set, or experience to handle a high tech company that deals in data in a worldwide setting.
The most unfortunate thing is that he doesn't value the people who are running the company 24-7.
Now many of them are gone, and the survivors are too few, and lack knowledge of critical hardware and software. The survivors will begin to realize that the price they are being asked to save the company is higher than they are willing to pay, and they will start bailing out.
Unless some major recovery of personnel is initiated (highly unlikely), it's not a matter of if the implosion happens, but when.

[usonian]

30. I've posted elsewhere that he was turning it into a startup, but it's a damn utility.

startup people get rewarded handsomely (hopefully) with equity and stock options.
To ask salaried people to work the equivalent of 80 hour weeks is criminal.

Unfortunately, bad guys get away with shit faster than anyone can respond (though there are plenty of lawsuits filed already).
Labor law violations are countless.

Come to think of it I was sysadmin for a Fortune 500 company.
i ran the mail server, which was a BFD.
It had to be patched to do SEC-mandated auditing.
And other stuff.

[orthoclad]

31. "Naive idiot" or stoner

Honestly, his behavior/posts/emails make me think he's constantly stoned.

It's like having Cheech and Chong run a global company.

He even made his purchase price a reefer joke: fifty-420 per share.

Defintitely, there is more to this that that, but I think intoxication has a part.

[Ziggysmom]

29. Systems Analyst & Programmer for over 35 years. The one thing I hate MOST, wether working

in Fortune 500 companies or the Future 50..... the dang high turnover rate in the IT sector. When you lose tenured, experienced people, even the best contractors or documentation in the world can't save your ass when shit blows up. May Muskrat now feel that pain

[GoneOffShore]

36. Right - Even as a complete layman in regards to the tech aspects, I understood a lot

This is a nightmare that can and will happen.

[ancianita]

38. Isn't this true of most social platforms? It's the way capitalist platforms can fail. Great.

Thank you! A real eye opener!

[tinrobot]

41. The ones that seem most damaging center around security

With few employees left to watch the gates, what's to stop a hacker group from holding all of Twitter hostage?

Then again, I suppose Elon has already taken the company hostage.

[calimary]

42. Dayum. This is nuts!

Not a techie here. I’m just glad we have people here who are. Cuz I sure don’t.

[usonian]

44. Twitter's Broken Its Copyright Strike System, Users Are Uploading Full Movies

Chaos on the increase.
https://www.forbes.com/sites/paultassi/2022/11/20/twitters-broken-its-copyright-strike-system-users-are-uploading-full-movies/?sh=3ff532f37d3a
I didn't run into a paywall. Use this if you do.
https://archive.ph/iaQ89

Last night, it became apparent that Twitter’s automated copyright strike/takedown system was no longer functional. A user went viral for uploading the entirety of The Fast and the Furious Tokyo Drift in two minute chunks over a 50 tweet thread. While it’s offline this morning, here’s where things get weirder still:




It should be fairly obvious to anyone what kind of liability it opens Twitter up to if their copyright system is non-functional, and its newly limited pool of workers are going to need to manually hunt down infringers. Once media companies get wind of this, we could see Twitter hit with all sort of DMCA claims and potential legal issues if they can’t get a handle on this quickly. I’m picturing Disney content starting to be uploaded here and them going nuclear. (emphasis mine.)

Also, it should be noted that one of Elon Musk’s big ideas for Twitter Blue is to allow users to upload long, 40+ minute videos. That would be a nightmare if they can’t fix their copyright enforcement system, but it’s not clear that anyone there is working on this issue in any meaningful capacity barring suspending that one specific Tokyo Drift account.

[Emrys]

47. It's not just that people can upload videos, they can and do use apps like Downloaderbot

to copy and store the material tweeted. So Musk's currently built a piracy hub!

[niyad]

45. "Fifty Ways To Leave Your Lover" started playing in my head at the title.