What do you think?
Rate this book
The Car That Knew Too Much: Can a Machine Be Moral?
The inside story of the groundbreaking experiment that captured what people think about the life-and-death dilemmas posed by driverless cars.
Human drivers don't find themselves facing such moral dilemmas as "should I sacrifice myself by driving off a cliff if that could save the life of a little girl on the road?" Human brains aren't fast enough to make that kind of calculation; the car is over the cliff in a nanosecond. A self-driving car, on the other hand, can compute fast enough to make such a decision--to do whatever humans have programmed it to do. But what should that be? This book investigates how people want driverless cars to decide matters of life and death.
In The Car That Knew Too Much , psychologist Jean-François Bonnefon reports on a groundbreaking experiment that captured what people think cars should do in situations where not everyone can be saved. Sacrifice the passengers for pedestrians? Save children rather than adults? Kill one person so many can live? Bonnefon and his collaborators Iyad Rahwan and Azim Shariff designed the largest experiment in moral psychology the Moral Machine, an interactive website that has allowed people --eventually, millions of them, from 233 countries and territories--to make choices within detailed accident scenarios. Bonnefon discusses the responses (reporting, among other things, that babies, children, and pregnant women were most likely to be saved), the media frenzy over news of the experiment, and scholarly responses to it.
Boosters for driverless cars argue that they will be in fewer accidents than human-driven cars. It's up to humans to decide how many fatal accidents we will allow these cars to have.
Human drivers don't find themselves facing such moral dilemmas as "should I sacrifice myself by driving off a cliff if that could save the life of a little girl on the road?" Human brains aren't fast enough to make that kind of calculation; the car is over the cliff in a nanosecond. A self-driving car, on the other hand, can compute fast enough to make such a decision--to do whatever humans have programmed it to do. But what should that be? This book investigates how people want driverless cars to decide matters of life and death.
In The Car That Knew Too Much , psychologist Jean-François Bonnefon reports on a groundbreaking experiment that captured what people think cars should do in situations where not everyone can be saved. Sacrifice the passengers for pedestrians? Save children rather than adults? Kill one person so many can live? Bonnefon and his collaborators Iyad Rahwan and Azim Shariff designed the largest experiment in moral psychology the Moral Machine, an interactive website that has allowed people --eventually, millions of them, from 233 countries and territories--to make choices within detailed accident scenarios. Bonnefon discusses the responses (reporting, among other things, that babies, children, and pregnant women were most likely to be saved), the media frenzy over news of the experiment, and scholarly responses to it.
Boosters for driverless cars argue that they will be in fewer accidents than human-driven cars. It's up to humans to decide how many fatal accidents we will allow these cars to have.
176 pages, Hardcover
Published October 12, 2021
5 people are currently reading
116 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 5 of 5 reviews
November 24, 2021
This slim book is unusual in taking us through the story of a single scientific study - and it's very informative in the way that it does it. The book makes slightly strange reading, as I was one of the participants in the study - but that's not surprising. According to Jean-François Bonnefon, by the time the book was published, around 100 million people worldwide had taken part in the Moral Machine experiment.
The idea behind the study was to see how the public felt self-driving cars should make what are effectively moral decisions. Specifically, in a dilemma where there was a choice to be made between, say, killing one or other person or groups of people, how should the car decide? As a concept, Bonnefon makes it clear this is a descendent of the classic 'trolley' problem where participants are asked to decide, for example, whether or not to switch the points so a tram that is currently going to kill five people will be switched to a track where it will kill one person who wouldn't otherwise have been harmed.
The many variants of the trolley problem are hard enough to deal with, but as Bonnefon makes clear, there are more complex determinants when dealing with autonomous vehicles. Apart from anything else, we would have to tell the car how to decide between killing its passenger(s) and a pedestrian. Would anyone buy a car if they were conscious that it would take the choice to intentionally sacrifice them where necessary?
Bonnefon and his colleagues decided to go beyond the usual scope of a study like this (which often just asks a few hundred students) to try to address humanity at large with a simple online system that presented each participant with a short set of preferences between, say, killing a jaywalking child or mounting the pavement and killing an elderly person who just happens to be passing.
In part, this is the story of how such a project works, giving rare insights into what researchers do, down to attempting to get into the big name journals and the fun of getting a bad peer review. This is really interesting, though in an attempt to make the story more approachable, Bonnefon does give us too much detail of the characters of his colleagues and where and how they met. Central to the book, though, is what might be done about autonomous vehicles. We are told of a German commission's findings on the subject, which is then contrasted with the findings from the study. (The main disparity is that the commission felt that cars should not be discriminatory, where the public at large thought they should be ageist.)
There are some omissions. When the results were published, one of the criticisms Bonnefon and colleagues got was that they asked people to make choices based on information that autonomous vehicles could never access - for example, choosing between a professional and a homeless person, a distinction the car couldn't possibly make. This seems a bizarre thing to do - surely the study should have been limited to choices that a car could actually make, and Bonnefon does not adequately defend the decision to include these extras.
The book also omits one of the key factors that could influence self-driving car uptake. Bonnefon spends quite some time on the idea that if the cars were only 10 per cent safer than a human driver, then anyone who thought they were at least 10 per cent better than average (which is most people) wouldn't want one. However, he didn't go into the potential problem that if autonomous cars are killing thousands of people, that would be thousands of real families seeing the negative side (and probably suing manufacturers), but the lives saved are statistical, so have nowhere near the same weight. It seems likely that for statistical benefits to outweigh damage caused (as is the case with a vaccine) autonomous vehicles would have to save many more lives than they kill.
There is a very interesting mention of the highly negative reaction received by a car manufacturer's representative, who pointed out that it may be better to definitely save a passenger when weighed against just a possibility of saving a pedestrian, but again Bonnefon doesn't adequately address this very reasonable view. The same issue comes up with many of the analyses of the trolley problem I've seen, where participants will usually switch a trolley to a different track with a remote switch, killing an individual, but won't push an individual off a bridge to their death if it would stop the trolley. This is put down to the remoteness of throwing the switch, but what hardly ever gets mentioned is that there can be a lot more certainty about the effect of throwing the switch compared with pushing someone off a bridge and hoping they somehow stop the trolley.
I would, then, have liked to have seen more detail here - but it's an enjoyable and easy read that both opens a window on what it's like to run such a study and gets the reader thinking about this very real decision that would have to be made if autonomous vehicles are to be allowed free access to our roads (something I suspect won't happen for a long time to come).
The idea behind the study was to see how the public felt self-driving cars should make what are effectively moral decisions. Specifically, in a dilemma where there was a choice to be made between, say, killing one or other person or groups of people, how should the car decide? As a concept, Bonnefon makes it clear this is a descendent of the classic 'trolley' problem where participants are asked to decide, for example, whether or not to switch the points so a tram that is currently going to kill five people will be switched to a track where it will kill one person who wouldn't otherwise have been harmed.
The many variants of the trolley problem are hard enough to deal with, but as Bonnefon makes clear, there are more complex determinants when dealing with autonomous vehicles. Apart from anything else, we would have to tell the car how to decide between killing its passenger(s) and a pedestrian. Would anyone buy a car if they were conscious that it would take the choice to intentionally sacrifice them where necessary?
Bonnefon and his colleagues decided to go beyond the usual scope of a study like this (which often just asks a few hundred students) to try to address humanity at large with a simple online system that presented each participant with a short set of preferences between, say, killing a jaywalking child or mounting the pavement and killing an elderly person who just happens to be passing.
In part, this is the story of how such a project works, giving rare insights into what researchers do, down to attempting to get into the big name journals and the fun of getting a bad peer review. This is really interesting, though in an attempt to make the story more approachable, Bonnefon does give us too much detail of the characters of his colleagues and where and how they met. Central to the book, though, is what might be done about autonomous vehicles. We are told of a German commission's findings on the subject, which is then contrasted with the findings from the study. (The main disparity is that the commission felt that cars should not be discriminatory, where the public at large thought they should be ageist.)
There are some omissions. When the results were published, one of the criticisms Bonnefon and colleagues got was that they asked people to make choices based on information that autonomous vehicles could never access - for example, choosing between a professional and a homeless person, a distinction the car couldn't possibly make. This seems a bizarre thing to do - surely the study should have been limited to choices that a car could actually make, and Bonnefon does not adequately defend the decision to include these extras.
The book also omits one of the key factors that could influence self-driving car uptake. Bonnefon spends quite some time on the idea that if the cars were only 10 per cent safer than a human driver, then anyone who thought they were at least 10 per cent better than average (which is most people) wouldn't want one. However, he didn't go into the potential problem that if autonomous cars are killing thousands of people, that would be thousands of real families seeing the negative side (and probably suing manufacturers), but the lives saved are statistical, so have nowhere near the same weight. It seems likely that for statistical benefits to outweigh damage caused (as is the case with a vaccine) autonomous vehicles would have to save many more lives than they kill.
There is a very interesting mention of the highly negative reaction received by a car manufacturer's representative, who pointed out that it may be better to definitely save a passenger when weighed against just a possibility of saving a pedestrian, but again Bonnefon doesn't adequately address this very reasonable view. The same issue comes up with many of the analyses of the trolley problem I've seen, where participants will usually switch a trolley to a different track with a remote switch, killing an individual, but won't push an individual off a bridge to their death if it would stop the trolley. This is put down to the remoteness of throwing the switch, but what hardly ever gets mentioned is that there can be a lot more certainty about the effect of throwing the switch compared with pushing someone off a bridge and hoping they somehow stop the trolley.
I would, then, have liked to have seen more detail here - but it's an enjoyable and easy read that both opens a window on what it's like to run such a study and gets the reader thinking about this very real decision that would have to be made if autonomous vehicles are to be allowed free access to our roads (something I suspect won't happen for a long time to come).
April 1, 2022
Good book about combining the trolley problem with autonomous, self driving cars (if the car can't save everyone, how is the car to be programed to choose who dies?). Also interesting look behind the scenes of publishing papers in major academic journals. (And also dealing with the press.)
January 14, 2022
Short interesting book on the ethical dilemma of driverless cars and deciding what lives they should be programmed to prioritize. It’s an interesting debate and one day these questions will need to be answered. Or climate change will change it all and the idea of having private transport vehicles will seem a wild luxury as we flee to the small remaining tracts of live-able land!
June 1, 2022
Cool book to read on the ethics of a scientific study. I especially liked the portions talking about how to deal with press.
October 17, 2022
Esta es la historia de los investigadores que comenzaron a preguntarse sobre la ética de los coches autónomos, tales como el problema del tranvía. Para ello, crearon la web MoralMachine, en la que se plantean distintos escenarios que podría vivir este tipo de vehículos, y cualquier persona puede responder. Se trata de tener una muestra enorme de respuestas y tendencias a lo largo del mundo, según edades, región, etc.
El libro es muy ilustrativo, tanto para el que le suenen estos temas como a los que no. Además, a pesar de ser un poco técnico, se lee y entiende perfectamente. Por todo ello, 4⭐.
El libro es muy ilustrativo, tanto para el que le suenen estos temas como a los que no. Además, a pesar de ser un poco técnico, se lee y entiende perfectamente. Por todo ello, 4⭐.
Displaying 1 - 5 of 5 reviews