Mere Goodness

Rationality: From AI to Zombies #5

by Eliezer Yudkowsky

Mere Goodness is the fifth book contained in the ebook Rationality: From AI to Zombies, by Eliezer Yudkowsky. It focuses on the relationship between moral theory and moral practice.

Mere Goodness contains three sequences of essays, along with the stand-alone essay Twelve Virtues of Rationality. These are all collected in the Rationality: From AI to Zombies ebook, but the essay names below are also linked to the original blog posts.

The previous book in the series is Mere Reality, and the next (and final) is Becoming Stronger.

Genres: Nonfiction, Philosophy

213 pages, Unknown Binding

First published January 1, 2015

About the author

Eliezer Yudkowsky is a founding researcher of the field of AI alignment and played a major role in shaping the public conversation about smarter-than-human AI. He appeared on Time magazine's 2023 list of the 100 Most Influential People In AI, was one of the twelve public figures featured in The New York Times's "Who's Who Behind the Dawn of the Modern Artificial Intelligence Movement," and has been discussed or interviewed in The New Yorker, Newsweek, Forbes, Wired, Bloomberg, The Atlantic, The Economist, The Washington Post, and many other venues.

Reviews

[Hilm · Rating 3 out of 5]

262 Dreams of AI Design:
Planes would fly just as well, given a fixed design, if birds had never existed; they are not kept aloft by analogies.

263 The Design Space of Minds-in-general
Any two AI designs might be less similar to each other than you are to a petunia. Asking what “AIs” will do is a trick question because it implies that all AIs form a natural class.

277 High Challenge
Timothy Ferris is worth quoting: To find happiness, “the question you should be asking isn’t ‘What do I want?’ or ‘What are my goals?’ but ‘What would excite me?’”

278 Serious Stories
This was George Orwell’s hypothesis for why Utopia is impossible in literature and reality: It would seem that human beings are not able to describe, nor perhaps to imagine, happiness except in terms of contrast

289 Something to Protect

In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif: Power comes from having something to protect. I’m not just talking about superheroes that power up when a friend is threatened, the way it works in Western fiction. In the Japanese version it runs deeper than that.

In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around.

[ROLLAND Florence · Rating 4 out of 5]

This book provides the reader with a baseline for a lot of the things that fascinated the LessWrong community in the following years:

* decision theory
* utility functions
* value theory
* what constitutes "good" and "altruism", what type of resource allocation makes sense for humanity
* game theory
* utilitarianism
* a pinch of AI, and some references to "friendly AI" even!
* some more bias and social psychology, because we are reading LessWrong after all (at the time when the Sequences were written, the blog was called Overcoming Bias).

If you became interested in the crypto disaster of Sam Bankman-Fried / Caroline Ellison and the rest of FTX, a lot of things will make sense here. I find it strange that Mere Goodness has so few reviews, because it really sets the scene for effective altruism. If you need one reason to read the book, this is a good one... Where does EA come from? What are the scientific / psychological / philosophical pieces of the puzzles, and which authors were leveraged to back those theories?

NB: I am not a believer in EA, not in the way that it was followed in the Silicon Valley at least. But this movement had so much impact on crypto AND on the economics of the 2020' that it is useful to understand its genesis.

[James · Rating 5 out of 5]

Less about ethics and meta-ethics than I expected, more about decision theory and biases than I expected. EY makes some important points in this book - the orthogonality thesis, an intuitive way to understand the real problem with the Prisoner's Dilemma, what I think is moral anti-realism, and the idea of "ethical injunctions" (which I call moral robustness). All important stuff, but not quite on the level of books 1 and 3. Looking forward to Becoming Stronger though.

Notes:
• It's really hard to properly do values -> goals -> actions. People tend to fit their values to the goal, and their goals to their actions. Doesn't always flow the correct way.
• Once again, knowledge is not some divine quest, or an puzzle the universe has set out for us, or really a part of the external "universe" at all. It's a mental tool - a set of magic spells we use to predict the future.
• The typical formulation of the Prisoner's Dilemma doesn’t really get across how much of a Dilemma it is. Humans are not truly selfish because we're wired to be empathetic and prosocial, so we naturally see D/D as the "best" configuration. EY proposes the more accurate way to see the prisoner's dilemma is when playing against a foreign intelligence, like a paperclip maximizer - that way H/D seems like the ideal outcome, and the real Dilemma becomes clear.
• Ok so I think EY is a moral anti-realist like I thought, he just doesn't describe himself that way. He says there is not universal morality in the sense of a "light shining from beyond", but uses "morality" as a term to basically mean the human utility function.
• Even not explicitly taking into account the full implications of moral uncertainty, robustness is still valuable, and agreement with societal moral views is still valuable. "You should never, ever murder an innocent person who’s helped you, even if it’s the right thing to do; because it’s far more likely that you’ve made a mistake, than that murdering an innocent person who helped you is the right thing to do."
• In general, explicit reasoning with probabilities should be rare, especially when the probabilities or Bayes factors have to be gut-estimated. This probably doesn't increase accuracy much, and makes you feel more Bayesian than you actually are.
• Numeric fallacy (my term): using numbers as a rhetorical device, not as numbers. Saying 1% chance when really all you mean is small chance. Subset of scientism fallacy (also my term), using science-associated concepts as rhetoric
• EY idiom - "'Rationality' is just the label I use for my beliefs about the winning Way—the Way of the agent smiling from on top of the giant heap of utility. Currently, that label refers to Bayescraft"
• The goal of rationality is not to be rational, but to win. Thinking about how to be rational is a proxy