This past week, one tiny mistake in the code of a single third-party provider, one designed to secure software, was released without proper staging. Over the course of hours that turned to days, vast swaths of the civilized world stopped working. Flights were grounded. Deliveries stopped. Bank software collapsed. Security systems froze. Even many electronic doors stopped working. All over the world.
It was the nightmare long predicted and long dreaded.
Personal computer systems were mostly unaffected, particularly those running iOS and Linux. The problem was networked computers and the software they were running from third-party provider CrowdStrike. The company sold the software as protection against cyberattacks. The software became the attack.
“The problem is that the technological revolution as we fashioned it 30 years ago gradually evolved in an ever more centralized way, wholly dependent on a weak and old-fashioned electrical grid of networks without much duplication or backstopping. The software, too, has become centralized for each industrial purpose. If one thing goes wrong in any system with a single point of failure, the whole comes to a grinding halt.”
Plus:
“It’s a terrifying thought that the whole of modern life hinges on such a thin foundation that can crack at any time, wholly changing reality in front of our eyes, taking down whole sectors, and disabling all functionality. ... Any system can be hacked and compromised in any sector: car sales, real estate management, delivery systems, banking and finance, and payment processing. It can all be here today and gone tomorrow. All these systems claim to have redundancies, but we have no guarantees of that. And we’ll never really know until they are really tested. Redundancy is just a management slogan. It might be real but most likely is not. In fact, there have been very few serious stress tests of anything built over the past several decades. We’ve just barreled ahead, piling digit upon digit and trusting that everything is going to work just fine forever. We have no assurance of that.”
What happened just a few weeks later was the anticipated nightmare.
All of this began decades ago with the obsession with creating vast networks, made necessary because Windows machines simply could not develop enough software to meet the market demand. Trust was high—I certainly trusted—and the networks grew and grew, within firms and industries and all over the world. A popular new software would be widely adopted by industry and maintained by the provider.
Most of these related to end users, but the real money was in software governing the administrative structure: security systems, cloud services, and other efficiencies. If they break, no user can fix them. They always require centralized fixes, geeks with passwords and secret forms of access. This is all because of security, don’t you know, and no normal person could ever be trusted to fix one’s own machine, much less update it.
Quick story from my personal work. It was perhaps 20 years ago when I was charged with upgrading all the office systems. I tapped a provider who immediately started building what is called an intranet, an internal network with a local server (which eventually moved to the cloud). As I watched this going on, I witnessed several occasions during which people’s machines needed to be updated but the user could not do it.
The provider immediately clarified that it would be charged with such a job.
I instantly pulled the plug on the whole scheme. Instead, I insisted on free-standing, independently operating machines. There was no need for anything else since this was not a bank or a financial firm but merely an educational service. It struck me instantly that any other system would face constant breakages with aging hardware, software updates, security threats, and so on. The scheme struck me immediately as a huge racket to provide revenue to outside companies.
Sure, sometimes it is necessary, but there was never any question about the problem. The issue is that a small breakage would bring down the whole. Now, back in those days, such systems were constructed only within a firm or institution. Over time, it became industry-wide and then went up and down the supply chains. The approach ended up creating gigantic hierarchies of control in which only digital oligarchs possessed all control.
That works until it does not. When it does not, the result is catastrophic. It seemed obvious to me then—and still does—that we do not want vast digital networks in which a tiny breakage would render millions and billions of people helpless as they sit and stare at blue screens of death. There was NEVER any question that this could happen. It was just a matter of when.
And by the way, Apple took a different direction, centralizing all core software and only reluctantly allowing applications with zero administrative access. Apple is usually behind the times in its software, a fact about which a generation of techies used to make fun. But no one is laughing now. Those systems were never threatened.
The frustrating feature of what happened last week is that nothing will change. Sure, the third-party providers will do every manner of internal investigation and swear to change. There will be loads of reports issued, and everyone will pretend to build redundancies and run tabletop games to prevent the previous error. None of this will stop the next problem because it will be different.
The multitudes of large corporations that rely on huge technical hierarchies will keep them in place. They will keep using radically insecure Windows systems and trust providers with fancy logos and large market capitalizations. And there will be more and more breakages, and they will be longer and more devastating.
Nothing can change this.
I’m going to say something no one wants to hear. The world converted from analog to digital technology too quickly, without putting in safeguards against failure and with an eye to sustainability. It was a frenzy to rebuild the world, replacing steel and bricks with gum, tape, tissues, and gassy hopes of some cloud-based reality no one had ever seen.
It was made possible only via cheap credit, fiat money, government subsidies, and high-flying financial markets, without which none of this would have happened.
Now we find that the whole of life depends on systems that no one in particular controls, that have never really been tested, and that cannot be fixed. The possibility of global breakage for an unlimited amount of time is very real. Think of it: Even our lights and air systems are networked and running on apps, to say nothing of the whole of the fiat-based and debt-driven monetary and financial systems themselves.
And yet here we are, with all the capital and energy being designated to building even more such systems, this time being run by artificial intelligence. Again, they are being constructed not as strong by virtue of decentralization but weak because of their central points of failure. As the parable says, these are houses built not on stone but on sand.
The world will rue the day this switch was made without thinking. This is already happening right now. I know exactly what my Mennonite friends would say about all of this. It looks ever more like they might be correct.
