Otherness and Control in the Age of AGI

by Joe Carlsmith

This book delves into the complexities of otherness and control in the context of artificial general intelligence, offering valuable insights on how these concepts impact our understanding and management of AGI. It explores the ethical, philosophical, and practical dimensions of AGI, making it a crucial read for those interested in the future of technology and society.
Cover painting: “The Creation” by Lucas Cranach

Genres: Philosophy

192 pages, ebook

Published June 18, 2024

Reviews

[Raoul G · Rating 5 out of 5]

"AI risk is substantially... a generalization and intensification of the sort of 'balance of power between agents with different values' problem we already deal with in the purely human world."

This is a collection of essays by Joe Carlsmith in which he philosophizes about our future with AGI. Especially he writes about different ways of relating to AGI (artificial general intelligence), especially through the lens of the way humans have generally related to Otherness, be it animals, other humans sufficiently different or aliens (in works of fiction). The other main aspect of the essays is control, which might in the future, in the age of AGI, not be what we as humans might have anymore - or at least not the most. As many people in the field, Carlsmith is concerned about existential risk from artificial intelligence - sometimes also called just AI risk. He engages with different perspectives but focuses mostly on the work of Eliezer Yudkowsky.

Carlsmith is a rigorous thinker, and a gifted writer. At times he draws inspiration from different movies and literature for analogies, which helps a lot with the understanding of the sometimes quite complex ideas. In general, I would say that the essays are geared towards readers with some background knowledge about the AI safety discourse.

[Idil Sahin · Rating 5 out of 5]

I never do work in planes and I read this holding my breath for four hours straight (and then more). The two seats next to me were empty, and I had to do a lit review for my lab meeting, and I had paid for airplane internet for the first time in my life. I did none of this. I read this book.

Joe Carlsmith does something remarkable here via making a parable of the way one interacts w the world/raises a child/works on alignment not through creating a manifesto but asking for your hand and taking you to check out your head being chewed out by a bear, meet neighbors who get off by staring at a paperclip swamped backyard, the color green, poetry and silence of the stars. His frame of morality is an invitation to experience, with kindness and strength and value/virtue as something created/discovered through interacting with the world. When he lets go and says goodbye, your hand feels warm, alive.


This is perhaps a sign that I do not read (or do) enough, but to see Ursula K Le Guin here as centerpiece made me care instantaneously, instead of jumping through 2893403901 hoops to convince myself daily Yes of Course I Care about AI not killing us all I work on this. Reading this also changed my mind about moral realism and turned a switch in my brain something in my brain where I now feel 10% happier.

[John · Rating 4 out of 5]

This collection of essays develops a set of ideas around the prospect of AGI and how we as humans should handle it. The essays vary in topic and depth. A lot of it focuses on responding to Yudkowskyian style thoughts around AGI, and so it has some of that flavor of LessWrong content, though in a more elegant and humanisticly educated style.

But a few of the essays are more tangential to AGI, and these, I think, are the better works in the collection by far. In particular, the essay On Attunement is a beautifully written meditation that reaches for, and I think finds, something quite deep about human experience and the universe. As with many of the best writings, On Attunement is itself an instantiation of its subject, and there is no higher praise I can offer.

On Attunement is a masterwork, and its well worth reading the whole collection just to have the full context for that one essay.

4/5 because On Attunement is 5/5 and much of the rest is nearer a 3 (which on my scale is still pretty good)

[!-!-!]

Articulates a set of insights I've gathered from a rather similar canon to the author. (He's missing Dostoyevsky, I haven't read Angels in America, but we agree on Annie Dillard, Martin Buber, and Le Guin.) Of course the problem is in the actual practice, especially as I am inclined to neither stillness nor empathy.
It's difficult to see how this practically translates to alignment, unless the answer is 'try the liberalism again'.

[Vivianne Manlai · Rating 3 out of 5]

Strangely, the content of the essay series sits inbetween analytical and continental philosophy. There are too many references to pop culture and made up moral concepts (yin and yang, green according to [insert other colour]) for me to find this particularly action-guiding or helpful.

Yet, I enjoyed the essay series. The last essay was especially touching.

[Aron Malmborg · Rating 5 out of 5]

Important. Beautiful. Profound.

[Michael Nielsen · Rating 5 out of 5]

A mind at work, trying to better understand how humans relate to the universe and to our creations. I loved it! It is uneven in places - it's an essay series, in some sense an extended set of notes on investigations, not a well-edited book serving a particular narrative purpose, and so some bits are rather tangential. But I really enjoyed the depth and earnestness of the underlying quest. And certain ideas - that of Deep Atheism, in particular - seem to me to be very important.

Edit: two months after finishing this, and I realize I've been thinking about it very frequently ever since, and it's changing my thinking (and beliefs) in important ways.