Spencer’s Reviews > The Unaccountability Machine: Why Big Systems Make Terrible Decisions - and How The World Lost its Mind > Status Update

In this slightly parodic picture of a big decision-making system, we have a compromised set of communication channels between black boxes, which is compensated for by adding resources in the shape of middle managers, who provide the capacity to translate the skewed signals from the financial reporting system into more or less accurate and actionable information.

The way that you deal with massively complex systems, according to the cyberneticians, is to split them up into black boxes with a manageable set of inputs and outputs. The combinations of those inputs and outputs will still be unmanageable, but that might not matter…
The important point is to get the description right in terms of black boxes, and to try to understand the relations between them.

The science fiction writer Charlie Stross, for example, described corporations as ‘very old, very slow AIs’ in 2017. A computer science professor called Ben Kuipers wrote a paper for a conference in 2012, in which he made the case that corporations met all the criteria necessary to be called independent beings, and that as such, they were artificial, intelligent and surprisingly successful in evolutionary terms.

The purpose of the pet hotel at Schiphol Airport was to shred squirrels; everyone involved may have cared about animal welfare, but the system had been set up to ensure compliance with import regulation at the lowest cost possible, so that’s what it did.

there is nothing to be gained by opening the black box. If you are trying to understand a system, all you can do is observe its behaviour, and whatever you can learn from doing so is all there is to know about it.

The principle of diminishing accountability: Unless conscious steps are taken to prevent it from doing so, any organisation in a modern industrial society will tend to restructure itself so as to reduce the amount of personal responsibility attributable to its actions. This tendency will continue until crisis results.

The people are brought into the story because that’s the only way to sensibly describe the systems they were part of. If it hadn’t been them, it would have been someone else. If this book had a real villain, we might identify it as ‘the regrettable tendency of complex systems to have opaque and volatile dynamics’.