Карта и местность

Rationality: From AI to Zombies #1

by Eliezer Yudkowsky

"If you live in an urban area, you probably don’t need to walk very far to find a martial arts dojo. "Why aren’t there dojos that teach rationality? "Very recently—in just the last few decades—the human species has acquired a great deal of new knowledge about human rationality. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. "These fields give us new focusing lenses through which to view the landscape of our own minds. We have a shared vocabulary in which to describe problems and solutions. "Humanity may finally be ready to synthesize the martial art of to refine, share, systematize, and pass on techniques of personal rationality." When human brains try to do things, they can run into some very strange problems. Self-deception, confirmation bias, magical thinking—it sometimes seems our ingenuity is boundless when it comes to shooting ourselves in the foot. In Map and Territory , decision theorist Eliezer Yudkowsky asks what a “martial art” of rationality would look like, beginning with the basic fighting stance—the orientation toward the world that lets us get the most bang for our cognitive buck, that best positions us to understand and react to brains’ strange acts of self-destruction.

Genres: Philosophy, Nonfiction, Science, Psychology, Self Help, Artificial Intelligence, Economics

ebook

First published January 1, 2015

About the author

Eliezer Yudkowsky is a founding researcher of the field of AI alignment and played a major role in shaping the public conversation about smarter-than-human AI. He appeared on Time magazine's 2023 list of the 100 Most Influential People In AI, was one of the twelve public figures featured in The New York Times's "Who's Who Behind the Dawn of the Modern Artificial Intelligence Movement," and has been discussed or interviewed in The New Yorker, Newsweek, Forbes, Wired, Bloomberg, The Atlantic, The Economist, The Washington Post, and many other venues.

Reviews

[Martynas Petkevičius · Rating 4 out of 5]

This first volume of Rationality: From AI to Zombies is a semi-organised collection of cognitive biases that, according to the author, often prevent us from finding out the truth. I would argue that it's hard to tell whether such biases actually deceive us in real life since they are only studied in unnatural experimental settings. Maybe these modes of thinking get us to the truth most of the time in daily life, but can be tricked into false beliefs by clever experiments, same way optical illusions trick us. I wish Yudkowsky addressed the limitations of social sciences.

I'd like to single out two cognitive biases from the book. The first one is the illusion of transparency, i.e. the tendency of a speaker to overestimate how much their listeners understand. Another one is the expectation of short inferential distances, which means that people often believe that some explanation requires only a few prerequisites, while in fact one may first need to learn a lot of other concepts for the explanation to make sense. I think Yudkowsky falls for both of these biases, because he expects the readers to already be familiar with Solomonoff's induction, Bayesian inference, many-worlds interpretation, etc. Overall, Map and Territory is a short and entertaining read, but the sections where the author jumps into maths assuming the reader knows all about it is a slog.

Sometimes it seems to me the authors leans too much towards psychoanalysis, trying to guess what exactly some speaker meant in a discussion. Like a true social scientist he comes up with or uses pre-existing names for reasons of belief: belief in belief, belief as attire, belief as cheer, professing, etc. I couldn't find a meaningful difference between some of them or use for the others.

I'm glad Rationality: From AI to Zombies is finally in print format. The cover has an interesting texture that feels like rubber (and attracts smears like a magnet), though, the binding quality is sub-par with pages bending for some reason. I just thought I'd mention this since aesthetics of a physical book is important to me.

I still rather liked the book and I'm sure most of my criticism can be attributed to my own ignorance. Still, I'll have new ideas to consider whenever making decisions or trying to determine the truth of things. I'd especially recommend Map and Territory to those better acquainted with reasoning, information theory and maths in general.

[Dmitry Loktev · Rating 5 out of 5]

A rational person, no matter how out of their depth they are, forms the best beliefs they can with the evidence they’ve got. A rational person, no matter how terrible a situation they’re stuck in, makes the best choices they can to improve their odds of success.

I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’.

Цепочки. Книга первая…
Звучит как начало Библии, да ещё произведение напичкано притчами и метафорами.

Итак, для начала это сложное чтиво. Я читал первую книгу год назад, сейчас же при необходимости написать отзыв обнаружил, что я не помню ничего конкретного. Отчасти это из-за сложности материала, отчасти из-за того, что я параллельно читал много статей про рациональность на lesswrong. Разные источники смешались в памяти и было сложно сопоставить какую идею\понятие я взял из книги, а какие из просто чтения сайта lesswrong или сопутствующих материалов.

Я не зря написал про Библию в начале. В тусовке московских рационалистов к «цепочкам» часто отсылают, если неофит задаёт какой-то вопрос. Посыл такой: «Иди читать цепочки, там намного подробнее и качественнее дан ответ на твой вопрос». Цепочки - Библия в том смысле, что они дают новый, не побоюсь слова «правдивый» взгляд на мир. Они отвечают на критически важные вопросы для жизни человека, восприятия вселенной, отношения к окружающим, правде и вымыслу.

Книга очень глубока. Автор имеет привычку нравоучительствовать, хотя в предисловии к книге он сам об этом извиняется. Мол цепочки – это сборник статей его сайта, а когда он их писал ему было 26-28 лет, молод был, горяч. Сейчас бы с дидактической точки зрения писал бы по-другому. Он и написал по-другому, намного более приятно для восприятия, но позже: Гарри Поттер и методы рационального мышления. Я ещё не дошёл, к сожалению.

So what, о чём книга?

Тяжело будет уместиться на одну, две, три страницы. Даже тяжело будет формулировать мысли о содержании книги, но я попробую. В книге 4 части.

1 Predictably wrong.
Объясняется концепт рациональности, что она есть и что она нет. Рациональность бывает эпистемологическая и инструментальная. Эпистемологическая рациональность – это степень соответствия карты (состояния мозга, системы убеждений о мире) территории (то есть реальности). Быть рационалистом в эпист. плане означает приходить к всё более аккуратным оценкам относительно того, что есть мир и как он работает. Huge. Задачка на всю жизнь.

Зачем строить аккуратную карту? Правильные модели обычно дают лучшие предсказания о будущих опытах, чем ошибочные модели.

Инструментальная рациональность – это способность рассуждать таким образом, чтобы в каждом случае наилучшим образом достигать целей.

Почему быть рационалистом здорово? Это не скучно? Здесь не надо много считать сложных математических уравнений? В общем случае не обязательно. Но быть рационалистом – значит обладать определённым набором знаний о биологии, эволюции, нейрофизиологии, математике, логике, психологии, чтобы лучше понимать прежде всего, как работает человеческий мозг по умолчанию, то есть интуитивно, если его не учить специально «думать лучше».

Почему рационалистом быть важно? Потому что рационалист быстрее понимает, как манипулировать миром. Люди заметили, что определённая модель мышления помогает быстрее находить взаимосвязи и достигать целей (ака выигрывать в терминологии lesswrong).

But what set humanity firmly on the path of Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world.

Во-вторых, первая часть раскрывает, что рациональность не есть противоположность эмоциям (я противопоставлял это до 28 лет). Рационалист не должен и не хочет «исключать эмоции из своей жизни». Ещё с курса я запомнил (очевидно, сука), что эмоции есть зажигание двигателя – они направление движения. Эмоции – источник действия. «Хотеть» - это эмоция. Как же это исключить!? Зачем?

При втором прочтении, я заметил, что эмоции – это также результат работы модели мира человека (т.е. системы убеждений). Наши эмоции проистекают от наших моделей мира. Бывает, что человек вырос, а модель мира у него осталась детская, подростковая или студенческая. Он уже во взрослом мире, а эмоции испытывать как ребёнок. Конечно, он же не перестроил механизм восприятия. Модель мира осталась прошлой, а нагрузки другие, данные на входе другие. Надо меняться.

И наконец, рационалист, если хочет – печалится, если хочет – кайфует. Иногда и часто от эмоций не скрыться – они возникают в силу нашего воспитания, происхождения, детских моделей восприятия, которые надо вырубать не то, что топором, а трактором проезжать по ним. Но рационалист в этом случае лишь использует свою эмоциональную реакцию как дополнительный фактор при принятии решения. Есть рацио, есть цели, есть эмоции. При выборе – нужно учитывать всё.

У нас очень американский подход: ты должен много пахать и заработать кучу денег. Мы про деньги и результат. И рационализм. Money talks, bullshit walks. А бирюлечки с пуфиками и смузи — понимаешь, это всё ролевые игры. (с) Олег Тиньков <3

Далее книга рассказывает о когнитивных искажениях. Ок.

И великолепная статья, фундаментальная: «ожидая коротких понятийных расстояний». Мне этого очень не хватало в Бауманском. Я ненавидел, когда препод по, скажем, комбинаторике, у доски, пропуская несколько математических действий, ставил очередное равно и говорил «из этого очевидно выходит, что…». Совсем не очевидно! Ты пропустил несколько понятийных расстояний!... Мог бы я кричать ему, если бы знал это понятие. Но тогда я просто сутулился ниже над тетрадкой и ругал себя внутренним диалогом за тупость.

2 Fake beliefs.
Опять фундаментальный жизненный концепт. Что есть убеждение? Какие убеждения стоит лелеять и охранять? От каких избавляться?

Юдковский придумывает смешную метафору: убеждения должны платить за аренду. Правильное убеждение – только то, которое имеет предсказательную силу, т.е. мы можем придумать эксперимент (опыт), выполнение которого подтвердит правильность нашего убеждения. Вообще идеально, если можно придумать гипотезу, реализация которой могла бы (в теории) опровергнуть убеждение.

Получается, что правильное (правдивое) убеждение – это такое убеждение, которое отражает реальность. А как проверить отражает ли оно реальность? Только на опыте. Только реальной жизнью.

Остальную часть главы Юдковский приводит примеры из жизни, когда люди говорят, что они верят (имеют убеждение), но это не убеждение, это fake belief. Он, разумеется, проходится по религии, политике, демократии и большом невидимом драконе в вашем гараже.

3 Noticing confusion.
Убеждения бывают сложными, многосоставными. Для таких чем больше результатов опыта, подтверждающих убеждение, чем лучше. И тут Юдковский задаётся вопросом, а сколько нужно результатов опыта, чтобы поверить? Результат опыта называется свидетельством. Сколько нужно свидетельств, чтобы поверить, что что-то истина?

Приводится разница между научным свидетельством, легальным и рациональным. Если я назову Андрея Пупкина убийцей, суд мне не поверит. Это не будет легальным свидетельством. Хотя будет рациональным. Мои слова имеют какую-то убедительную силу. Не достаточную для законодательства большинства стран, чтобы суд обвинил Андрея в убийстве. Если же я проведу научный эксперимент, проведу научные пробы ДНК, подтверждающие причастность Андрея, то это будет уже научным свидетельс��вом, которое может быть воспринято и легально.

Сила рационалиста в том, чтобы натренировать мозг замечать, когда свидетельства шепчут (когда кричат – легко) о несоответствии реальности и карты (убеждения). Часто для признания этого несоответствия нужна смелость. Особенно в любовных ситуациях. Особенно в молодости (личный опыт).

Что есть правда? Это убеждение (состояние мозга) результат эксперимента (желательно повторяемого) по которому совпадёт с предсказанием.

4 Mysterious answers.
Очень. Сложная. Глава. Через неё надо было в буквальном смысле продираться как сквозь густые джунгли. Загадочными ответами Юдковский называет совокупность типов ответов, которые семантически являются ответами, но не позволяют сформировать проверяемую гипотезу, которая будет обладать предсказательной силой.

- Почему идёт дождь?
- Потому что небесные духи дерутся, и кто-то проиграл.

Как мне, как рационалисту, помогает это объяснение? Что я узнал о мире? Что существуют духи? А откуда они взялись? Как я могу предсказать, когда дождь пойдёт в следующий раз?

Автор приводит такие ответы. Пример выше – это как раз fake explanation. Такое объяснение не даёт мне знания, чтобы сформировать проверяемую гипотезу, провести эксперимент. Такое убеждение не имеет ценности.

Другой тип поддельных объяснений – это семантические стоп-знаки. В культуре существует ряд слов и словосочетаний, произнося которые человек сам перестаёт желать докопаться до сути или перестаёт объяснять, подразумевая, что всё должно быть понятно.

Разумеется, самый популярный «знак остановки любопытства» - это религия.

- А как, а откуда, а почему?
- На всё воля Божья.

Другие curiosity stoppers – демократия, воспитание, «всё из детства» и т.д. Даже наука сегодня часто срабатывает семантическим стоп знаком, прерывающим необходимость поиска объяснений. Современный айфон для любого человека из 60х будет чистой магией. Если он его увидел, возможно, подумал бы что на землю уже спускались инопланетяне, оставив свою технологию, или это пример земной магии. Он бы тут же стал страстно искать возможности разгадать, понять эту технологию.

Современный человек же не лучше понимает, что происходит, когда он включает свет в квартире маленьким рубильничком, когда нажимает «search» в гугле, когда отсылает смску. Но он не ищет ответов. Для современного представления размышление в духе «кто-то знает, значит я знаю» - норма. По сути, не происходит ничего другого, как то, что «мы верим», что это наука. Но мы уже не в состоянии это проверить, провести эксперимент.

Итак, автор заключает, что в каждом случае, когда мы сталкиваемся с таинственными ответами, у нас есть выбор между:
1. Искать объяснения.
2. Искать поклонения.
3. Проигнорировать.

Беда в том, что я сам и моё окружение часто тяготеем к 2 и 3.

Топ-3 мысли.
The strength of your belief should be it’s ability to predict experimental results.
Our emotions arise from our models of reality.
Always hit explain, don’t be satisfied with ignorance or worshiping (=watch out for curiosity stoppers).

I visualize the past and future of humankind, the tens of billions of deaths over our history, the misery and fear, the search for answers, the trembling hands reaching upward out of so much blood, what we could become someday when we make the stars our cities, all that darkness and all that light—I know that I can never truly understand it, and I haven’t the words to say.

Я думаю, что эта книга стоит того, чтобы её перечитать и в третий раз. И будет опять полезно. Мозг сложно и долго переучивается на то, что

[Carolyn Tragasz · Rating 5 out of 5]

This book is important because believing true things is important. While I am not a rationalist, philosophically, I agree with the thesis underlying these essays. The human brain has biases that prevent us from "seeing the territory" accurately. I think rationality works as a pragmatic tool, not as a worldview, and these essays are like a series of mental exercises designed to help train yourself to account for biases, so you can "level up" your reasoning skills, making better predictions and thus better decisions.

The essays, originally published as blog posts, are precise, witty and delightfully written, priortizing accessible content over intellecutal showmanship. Yudkowsky is a very effective communicator, and his parables especially stand out. The "Fable of Science and Politics" made me cry and other analogies made me laugh.

This book should be part of the curriculum in any logic and reasoning course, and can be re read like a series of sermons, worth pulling out and reviewing periodically.

[Jig · Rating 4 out of 5]

this paragraph captures why you should read this book:

what do i mean by "rationality"?
1. epistemic rationality: systematically improving the accuracy of your beliefs.

this is about building accurate maps that correspond to reality.

2. instrumental rationality: systematically achieving your values.

this is about steering reality -- sending the future where you want it to go

some chapters drag on but more or less required reading for clarity of thinking

[Dennis Nehrenheim · Rating 2 out of 5]

Context & Why I read this book
This is my third book in 2021; part of my challenging 52-book challenge that goes with my slogan "A Year of Ratio & Will". My motivation was to think, become, and act more "rational"; to become a "rationalist" to say it in the author's words.

What the book is about
The book is the first volume in a book series called Rationality: From AI to Zombies, which addresses our "flawed" psyche. In each chapter, a human bias, logical fallacy, or other shortcomings of the human mind is addressed. They are categorized into 4 parts which should give you a very rough feeling of the content:
1. Predictably wrong (e.g. confirmation and availability bias, planning fallacy, ...)
2. Fake Beliefs (e.g. applause lights, religion, retreat to commitment, ...)
3. Noticing Confusion (e.g. Occam's razor, hindsight bias, expecting short inferential distance, ...)
4. Mysterious Answers (e.g. fake explanations, "positive bias",
While some definitions and explanations are given, this book is not an encyclopedia of cognitive biases. Rather, it is a compilation of loosely connected essays that the author compiled over the years. This shows. The style and quality of these essays vary a lot and make this all in all a very sad read.

One lesson I am taking with me
There are a few nuggets to be found in this book. The one I like the most is his recommendation that you should keep asking yourself the following question: "How would I regenerate this knowledge if it were deleted from my mind?". This is to ensure that you are not fooling yourself into believing that you know the meaning/concept behind a certain term. Additionally, I added this quote "That which can be destroyed by the truth should be." P.C. Hodgell to my quote database.

Reading Recommendation / Who should read this?
Ironically, my review might be biased since my two previous reads were both on rationality (A Guide to Rational Living, The Enigma of Reason) and they were exceptionally good. But compared to them, this book very much feels like a concatenation of some cheap "blog posts" the author wrote in his spare time. So, I cannot agree with most of the reviews that came before me. Yudkowsky is not a very good writer. He appears more like an arrogant ranter. Most of the essays are rather confusing and pointing out how flawed we are in our thinking. So, considering there is an almost unlimited number of books on rationality you could read, I would not recommend picking this one and rate it a 4 out of 10 on my personal book rating scale which maps to Goodreads as follows:

1 ⭑ — Abysmal; extremely bad. Couldn't / wouldn't finish. No one should waste his time on this!
2 ⭑— Very bad; skipped part of it; skimmed most of it.
3 ⭑⭑ — Bad, but forced me to finish; close to no nuggets to be found.
4 ⭑⭑ — Rather bad; finished but will not give it a re-read.
5 ⭑⭑⭑ — Modest; a few nuggets; reading recommended in certain circumstances
6 ⭑⭑⭑ — OK; the average read. Tangible weaknesses, but recommended with some reservations
7 ⭑⭑⭑⭑ — Good read, despite minor weaknesses; generally recommended
8 ⭑⭑⭑⭑ — Very good; would recommend nearly without restriction
9 ⭑⭑⭑⭑⭑ — An outstanding work; worthwhile to be read twice or more often; a definitive recommendation
10 ⭑⭑⭑⭑⭑ — A work of genius; should be required for everyone

[Christopher Devine · Rating 5 out of 5]

Thought-provoking

[Demetre Saghliani · Rating 5 out of 5]

It's a one of a kind book, which makes it hard to rate; not many others of its type to compare to.

It's just a sorted collection of the authors essays, but it's done well and reads like a narrative. Worth reading for sure, but it's information-dense, so I recommend doing so over a longer period of time, rather than in, say, a week.

[Jonathan Ammon · Rating 3 out of 5]

I will have to return to this when I am actually determined to learn how to do proper Bayesian calculations. Unfortunately, actual mathematics have always bored me.

[Jacob Williams · Rating 3 out of 5]

But if you can’t say “Oops” and give up when it looks like something isn’t working, you have no choice but to keep shooting yourself in the foot. You have to keep reloading the shotgun and you have to keep pulling the trigger. You know people like this. And somewhere, someplace in your life you’d rather not think about, you are people like this.

Being a collection of blog posts, this is somewhat disjointed, and the level of assumed background knowledge seems to vary significantly among chapters. But it’s a nice grab-bag of information about biases and Bayesian reasoning.

[Ondřej Plachý · Rating 5 out of 5]

Absolutely essential compendium of essays by Yudkowsky. This is an essential read if you are interested in rationalism. Not really deep in philosophy which is for the better. Contains a pretty good introduction into Bayesian probability.

[Tarmo Pungas · Rating 4 out of 5]

Introduction to rationality mainly focusing on biases.

[Harry Taussig · Rating 5 out of 5]

# The Book in 3 Sentences

1. Being rational is a way to make our beliefs about the world more true, which matters so that we can better influence things in the ways we see fit.
2. We know about many of our biases, but this is not enough to overcome them, and we must be attentive and actively try to spot our biases in the moment or we will fall for them.
3. There are many ways we convince ourselves that we know more than we do, and we must repeatedly ask ourselves what the evidence is behind our believes, and what outcomes our beliefs predict in the real world.

# My Top 3 Quotes

1. "In the ancestral environment, anyone who says something with no obvious support is a liar or an idiot. You're not likely to think, "Hey, maybe this person has well-supported background knowledge that no one in my band has even heard of"
2. "If you don't have enough experience to regenerate beliefs when they are deleted, then do you have enough experience to connect that belief to anything at all?"
3. "If you can explain all outcomes equally well, then your belief doesn't actually tell you anything."

# Impressions

Very clear. Very clever. Dangerously clever in making yourself feel like you've overcome all the biases just by reading about them and the "stupid" people who fell for them. But the whole point is to not fall into those kinds of traps, since we all feel that we are probably less biased than others.

## Who Should Read It?

Anyone serious about writing, learning, or making a genuine impact on the world. True beliefs are essential to making a difference, as they are what we base our actions off of (unless we are choosing randomly) and we need to build active mental habits to help us be rational because we are certainly not by default.

# How the Book Changed Me

How my life / behaviour / thoughts / ideas have changed as a result of reading the book.

Recognition that just knowing about my biases is not enough to defeat them.

Another layer of disillusion in the ways I deceive myself, and a better understanding of where I am likely to do so in the future.

An appreciation for how hard and how much presence of mind it takes to begin to overcome these biases

Finally a genuine grasp of bayesian reasoning

[Dustin Blanton · Rating 4 out of 5]

Yudkowsky gives a primer on the fundamental elements of what he feels is the 'martial art of rationality', attempting to teach the reader self-defense against magical thinking.

It's a serious challenge, because as it turns out the human brain did not develop in an environment that promoted accuracy; instead, we are subject to a staggering array of biases that relate to things such as our tendency to believe statements made by members of our immediate in-group, and our complete lack of familiarity with problems of large scope.

What words and phrases do we use that are devoid of meaning? What concepts or ideas are actually mental 'stop-lights', or placeholders for understanding that cause us to skip over areas that require more thought?

A useful set of tools to have in the toolbox, though to acquire them one will have to wade through some of Yudkowsky's more pedantic and (in some cases) unnecessarily dogmatic moments.

[Alex Ager · Rating 3 out of 5]

A lot of great ideas in this book about rationality, which for me were overshadowed by the cringeworthy and arrogant writing style. It felt at times like I was reading posts from r/iamverysmart. Even still, I learned some things of value from this book, including a lot of depth about Bayesian reasoning/statistics. This was a good test run for the longer "Sequences" that Yudkowsky wrote in 2008-09, which run into the thousands of pages. I think this short volume will suffice for me. Especially given that writers I enjoy much more (Steven Pinker, Julia Galef) have recently taken to popularizing many of the ideas that Yudkowsky covers here.

[Niklas · Rating 4 out of 5]

Want to improve your thinking?
Start here.

This book (the whole series, really) offers a great meta-approach to thinking more clearly, logically and just less wrong.

But in order to debug the mind, it is essential to observe what consistently goes wrong first.
This book starts doing so by considering cognitive biases, but then also goes on to explain many more of the abundant failure modes of the human mind.

It also gives a good introduction on the concept of Rationality and explains what the scientific methodology actually is.

[Julissa Dantes-castillo · Rating 4 out of 5]

This book gives a pretty interesting insight in the way we see the world, some chapter were a little repetitive, and in other I didn't see a structure, but overall the book was fascinating. It is not a book that will change your life or anything like that, but presents many concepts that will help along the way.

[endofsilence · Rating 5 out of 5]

No other book I've read made me as less wrong as this one.

[Aliaksandr · Rating 3 out of 5]

Partially confusing, partially boring and partially interesting.

[Jamie · Rating 2 out of 5]

Helped me fall asleep at nights.

[James · Rating 5 out of 5]

Solid piece of work. Nothing revolutionary so far, I feel like I'm pretty familiar with most of the ideas EY focused on in this section. Still provided some interesting food for thought though, especially that neat little falsification/verification Bayesian thing at the bottom of my notes. Still waiting for my Philosophy of Science prof to respond to my email about that :(

Notes:
• Distinction between epistemic and instrumental rationality seems trivial to me. Epistemic rationality is a powerful tool, but still a subset of instrumental rationality.
• Building anti-flood dams prevents only small floods - this reduces flood precautions, and the availability bias is so strong that overall flood damage often increases
• Conjunction fallacy happens when adding detail makes a scenario seem more plausible, but necessarily makes it less probable. This is easy to detect in its simplest form, but more complex examples like large narratives with many added details can be harder to spot
• Rationally fixing biases is hard - it means you have to notice the fallacy every time. A better solution might be to construct new heuristics or adjust old ones, rather than notice bad heuristics and reason algorithmically. Solution has to be sub-verbal
• Planning fallacy - best-case and best-guess scenarios yield indistinguishable predictions. The default guess is no unexpected failures
• Back in the old days, saying the local religion “could not be proven” would have gotten you burned at the stake. One of the core beliefs of Orthodox Judaism is that God appeared at Mount Sinai and said in a thundering voice, “Yeah, it’s all true.” From a Bayesian perspective that’s some darned unambiguous evidence of a superhumanly powerful entity.
• The appearance of science/logic are very different things than actual science/logic. Science is a process, not a set of words. Technobabble is the same as magic spells, and scientific concepts are no better than god or any other blind dogma when severed from the actual scientific method.
• It is far better to say “magic,” than “complexity” or “emergence”; the latter words create an illusion of understanding.
• Interesting that Yudkowsky thinks emergence is nonpredictive/nonscientific, but likes GEB so much. I agree that emergence isn't really science, but to me, GEB's attempt to explain consciousness via "strange loops" fits right into this category
• 2-4-6 experiment: we are programmed with verification, not falsification, as the default scientific method
○ I guess that means verification works well enough most of the time, at least in our evolutionary history
○ What is the real difference here? If I think phlogiston will result in phenomenon X and test that way, why is that different than believing phlogiston theory prohibits !X?
○ There is no difference - clearly it's not about verification vs falsification. The difference is in which X I test. If I want to learn about the likelihood of a theory T from observing phenomenon X, then P(T|X) = P(X|T)P(T)/(P(X). If theory T predicts phenomenon X, then P(X|T) is 1 (or at least very close to it, depending on the experiment. That leaves only P(T|X) = P(T) * 1\P(X), so the initial P(T) is updated by a factor of 1/P(X). Powerful results come from surprising predictions (low P(X)).
○ This is the exact opposite of what current academic incentives point towards! Potentially revolutionary experiments should fail the majority of the time (negative, not null).

[Jason Curran · Rating 4 out of 5]

"Above all, don't ask what to believe - ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be at the center of inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it."

"Belief is easier than disbelief; we believe instinctively, but disbelief takes a conscious effort."

"Your strength as a rationalist is your ability to be more confused by fiction than by reality."

Absence of evidence is not evidence of absence.

Illusion of transparency - Our words are more ambiguous than we think. What I say may convey all of the necessary context in my own brain, but rarely do other people fully see the same picture.

Self-anchoring - The tendency to model other minds as though they were slightly modified versions of oneself.

"A clear argument has to lay out an inferential pathway, starting from what the audience already knows or accepts. If you don't recurse far enough, you're just talking to yourself." (this is also teaching)

"To worship a phenomenon because it seems so wonderfully mysterious is to worship your own ignorance." This isn't always bad, or even avoidable, but it's true.

Logical Fallacy of Generalization from Fictional Evidence - We're hard-wired to believe what we see with our own two eyes. Movies we've seen, history we haven't. But making predictions about the future should be more based on what has actually happened rather than by what fictional stories we tell. "Remember how, century after century, the world changed in ways you [could not have] guessed."

Our reaction to the unknown can take three forms: IGNORE/EXPLAIN/WORSHIP

"Look at yourself in the mirror. Do you know what you're looking at? Do you known what looks out from behind your eyes? Do you know what you are?...
Do you know how your knees work? Do you know how your shoes are made? Do you know why your computer monitor glows? Do you know why water is wet?
The world around you is full of puzzles. Prioritize, if you must. But do not complain that cruel science has emptied the world of mystery."

"How would I regenerate any knowledge if it were deleted from my mind?" -- Make sure you understand the thing, not just know the fact.
-- "When you contain the source of a thought, that thought can change along with you as you acquire new knowledge and skills. When you contain the source of a thought, it truly becomes a part of you and grows along with you.
Strive to make yourself the source of every thought worth thinking. If the thought originally came from outside, make sure it comes from inside as well."

[ROLLAND Florence · Rating 4 out of 5]

This is a first book of a pretty ambitious sum called Rationality: From AI to Zombies. It is actually a collection of blog posts from the blog LessWrong. Everything is available for free, online.

Now this is the typical book to read for an engineer working in tech. A lot of people interested in AI safety, cognitive bias and anything in between end up reading the Sequences. We can give credit to Yudkowsky for thinking about those topics *a lot* and writing about them as well, before it was trendy.

With that out of the way, I had read some of those articles, but never went on a mission to read all the Sequences. But hey, I decided to get started, so here we go. I will review all the books as I make progress.

This is easy to read IF and only IF you are already acquainted to Bayesian statistics. A good baseline knowledge of physics will also help you. I would say that this is psychology and ethics for engineers and scientists. It is not the kind of book that I would call "entry level". The style is a bit pompous. These articles remind me of the type of conversations that you sometimes have at the coffee machine in a tech company. The other person throws advanced mathematical concepts at you to see if you are worthy of their attention. Are you part of the in-group of tech / scientists / rational people? I guess that *understanding* the Sequences can make you feel clever (and learn some pretty useful things about the way). The problem is that in some cases, Yudkowsky sounds a bit like a pompous asshole with a big ego. "Hey, I was reading advanced physics as a teenager!" - we do not need this information to actually learn about cognitive bias, thank you.

Why 4 stars then? Because I actually learned some things, and the short format works for me. Those articles aged really well (they are more than 15 years old). The scientific references are on point. It is even fun to read. But I feel like the honest way to proceed would be to present the prerequisites, instead of pretending that this book is an introduction to the topic.

[Sebastian Nicolaisen · Rating 4 out of 5]

We never become completely right in science—we only become less wrong.

Now I’ve dug myself into the world of Rationality, and I’m not sure there’s a way back. Ever since I first encountered Daniel Kahneman in my youth, I’ve been searching and pondering what can truly be called “truth.” That path led me to Herbert A. Simon and his theory of bounded rationality—and eventually I stumbled into the writings of Eliezer Yudkowsky.

This book feeds my intellectual thirst and hunger. I can’t say I understand everything, but every now and then I catch glimpses of clarity, even as much of it sails far over my head.

Rationality, to me, is more about how you look at the world, and what reflects back onto your retina. What is true and what is not—I honestly don’t know. But at least I’ve begun my training in the martial art of rationality.

At times, the book feels heavy to chew on, almost drifting into philosophy, and I struggle to keep both feet on the ground. Maybe that’s because I’ve never thought so deeply about these questions before, and have mostly been trained to think in more traditional rational terms.

Is there such a thing as absolute truth?

Eliezer offers a provocative book (really a collection of blog posts) that provides both insight and a starting point.

[Chloe Duckworth · Rating 5 out of 5]

The first book of sequences is the closest cogent guide to thinking that I have read outside of formal logic texts. I think Yud is exceptional at explaining nuance without isolating readers, and by inviting us to join the act of thinking rather than feel intimidated by it.

I particularly enjoyed the sequences: Einstein’s Arrogance, Belief as Attire, Applause Lights, Religion’s Claim to be Non-Disprovable, Pretending to be Wise, Bayesian Judo, Belief in Belief, Making Beliefs Pay Rent, and A Fable of Science and Politics.

However, I found many particularly distracting logical fallacies in some of Yud’s arguments that kept me quite engaged in my interrogation of the novel. I think this is the way any rationalist should approach a text claiming to hold universal truths of thinking and reasoning. And it perhaps strengthens the message of rationalism that Yud unwittingly fell victim to a handful of lapses in rational thought, which engaged his readers to immediately practice the skills we’re tuning.

[Weltengeist · Rating 4 out of 5]

The series "Rationality: From AI to Zombies" is derived from a blog that the author has maintained for many years. It's goal is to teach the basics of rational thinking - maybe even to a non-technical audience (though I'm a bit sceptical that the author is aware of just HOW untechnical a non-technical audience can be).

The chapters (corresponding to blog entries) are thus often short (2-3 pages) and sometimes repeat older material. But they discuss a lot of material from numerous disciplines, and at least I am not aware of a coherent textbook on all these topics. Thus, I find familiar concepts alongside with thoughts I never heard about, and this made (at least some of) the text rather interesting.

With the sole exception of the final 20-pages chapter - I simply did not get what message the author is trying to convey there...

[Reed Schwartz · Rating 2 out of 5]

I always finish books but I DNF the rest of the sequences (read: I got filtered.) I didn't find anything in this book to be especially interesting or revelatory and his writing is incredibly annoying: it's like reading a whole book of posts on reddit in 2009. There are multiple instances of him transcribing conversations he had at conferences where he
1. totally owns the illogical rubes around him
2. looks like a huge asshole
(you can use your rationality to choose which map better corresponds with the territory.)
Maybe these pick up later but my Bayesian reasoning is telling me not to continue. Probability that I finish the rest of the sequences by 2030: 10%.

[Adina · Rating 5 out of 5]

Yudkowsky’s writing is sensitive and often funny. He ventures into “territory” not often associated with “cold” logic. Even as it pushes the reader to reconsider all of her biases and acknowledge the biases of others its approach feels like a balm in this contemporary moment in which it often feels like everyone is braced for intellectual and cultural combat.

[Tim · Rating 4 out of 5]

Collection of essays about how to think straight and avoid falling into errors. Mostly quite accessible and also short pieces. Pretty good. It has a theme but lacks an arc, but I suppose an arc isn't obligatory. Maybe it's an error to want arcs.

[Lior · Rating 4 out of 5]

Very interesting read. Recommended for any aspiring rationalist or anyone seeking to look more clearly into things.

[Kevin · Rating 4 out of 5]

A neat little encyclopaedia of cognitive biases and heuristics for how to overcome them. It written in the style of a self-help book with little parables to illustrate the ideas.