Software Runs the World: How Scared Should We Be That So Much of It Is So Bad?

[xchrom]

http://www.theatlantic.com/business/archive/2012/08/software-runs-the-world-how-scared-should-we-be-that-so-much-is-bad/260846/



What do most people think of when they think of software? A decade ago, probably Microsoft Word and Excel. Today, it's more likely to be Gmail, Twitter, or Angry Birds. But the software that does the heavy lifting for the global economy isn't the apps on your smartphone. It's the huge, creaky applications that run Walmart's supply chain or United's reservation system or a Toyota production line.

And perhaps the most mission-critical of all mission-critical applications are the ones that underpin the securities markets where a large share of the world's wealth is locked up. Those systems have been in the news a lot recently, and not for good reasons. In March, BATS, an electronic exchange, pulled its IPO because of problems with its own trading systems. During the Facebook IPO in May, NASDAQ was unable to confirm orders for hours. The giant Swiss bank UBS lost more than $350 million that day when its systems kept re-sending buy orders, eventually adding up to 40 million shares that it would later sell at a loss. Then last week Knight Capital -- which handled 11 percent of all U. S. stock trading this year -- lost $440 million when its systems accidentally bought too much stock that it had to unload at a loss.* (Earlier this year, a bad risk management model was also fingered in JP Morgan's $N billion trading loss, where N = an ever-escalating digit.)

The underlying problem here is that most software is not very good. Writing good software is hard. There are thousands of opportunities to make mistakes. More importantly, it's difficult if not impossible to anticipate all the situations that a software program will be faced with, especially when--as was the case for both UBS and Knight--it is interacting with other software programs that are not under your control. It's difficult to test software properly if you don't know all the use cases that it's going to have to support.

There are solutions to these problems, but they are neither easy nor cheap. You need to start with very good, very motivated developers. You need to have development processes that are oriented toward quality, not some arbitrary measure of output. You need to have a culture where people can review each other's work often and honestly. You need to have comprehensive testing processes -- with a large dose of automation -- to make sure that the thousands of pieces of code that make up a complex application are all working properly, all the time, on all the hardware you need to support. You need to have management that understands that it's better to ship a good product late than to ship a bad product on time. Few software companies do this well, and even fewer of the large companies that write much of their software.

[Warren Stupidity]

1. As you point out, it doesn't have to be crappy. For example: Curiosity.

But that was done outside of market constraints. To some extent the reason why much of the technology we use is really rather buggy is an example of market failure.

[xchrom]

2. curiosity is wonderful example of what i think of as 'institutional success'.

my own made up phrase for the opposite of what we all seem to experience too much of these days -- 'institutional failure'.

from crappy government responses to the needs of the people to the wall street debacle.

we really can be more successful -- but standards, rationality, good reasoning, science or evidence has to be our bedrock again.

[Warren Stupidity]

3. In the late 80's/early 90's, just before the .com boom

the computer industry had many large profitable companies that had what appeared to be market stability. There was a lot of focus on processes and quality in software development as a way to gain competitive advantages. Then the 'minicomputer crash' in the recession of the early 90s, marking the rise of the WinTel dominance of the market, destroyed that stability, vaporizing most of those companies in the process. We went into the .dot com boom with a whole new set of institutions with no history and with a philosophy summarized as "Don't Worry, Be Crappy". First to market was everything. Software and hardware (but primarily software) were delivered with Alpha level quality to customers, customers became the QA department. We haven't recovered from that mess.

[slackmaster]

4. The real problem is that many people have blind faith in the accuracy of electronic records.

Anyone who has ever tried to convince a bank employee or someone in health care that their records have inaccurate data knows this to be true.

[Ezlivin]

5. Have any of you ever written code?

If you have then you'll know that most development time is spent in debugging code, not writing it.

A lot of methods have been introduced to reduce or eliminate the need to debug code, but none of them have been successful.

The one thing that scared a lot of knowledgeable people was Reagan's drive to militarize space through his Space Defense Initiative. When it was revealed that over a million lines of code would be necessary the gig was up; it's just not possible to insure bug-free code of that length. While a million lines of code that contains a few bugs is merely bothersome when it's an office application, bugs in code that direct weapons is a whole different problem. With the high speed electronic trading that's now popular the same holds true. Will a bug in software bring down our markets or damage them so severely that it impacts the country?

[bemildred]

6. +1.

I've written code, lot's of code, and you are quite correct. "Zero defects" is possible, but it's very expensive, mainly because the sort of coders that can actually do that sort of work successfully are rare.

Your comment about the many fatuous attempts to make programming an unskilled position are well taken too. What you get with unskilled programmers is unskilled code, it is very much an accrued skill with no shortcuts, and that is why so much commercial software is buggy crap.