Friendship is Optimal

Friendship is Optimal

by iceman

Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.

Genres: Fiction, Science Fiction, Fan Fiction, Artificial Intelligence, Singularity, Fantasy, Philosophy

123 pages, ebook

First published November 14, 2012

Reviews

[Aaron Gertler · Rating 5 out of 5]

"Some readers might find that this story depicts a deeply horrifying end-of-humanity scenario, and other readers might consider it a triumphant ascent to a (slightly odd) utopia. I can see a little of both, really."

--Facedeer

I second the above. I'll add that I found the characters to be different and interesting enough to propel the story forward; the writing is better than that of all but the very best fanfiction.

It's remarkable how many things the author does that haven't often been seen in mainstream sci-fi: In addition to the step-by-step description of a takeover by a reasonable, friendly artificial intelligence, we also get the adjustment of an ordinary human to an AI-driven utopia, the exploration of a different system of physics from the one on Earth, a bunch of different reverse-AI-box conversations... and that's just the first half of the story.

Recommended for: Transhumanists; rationalists; people who hate transhumanism (this story sees it as an extreme moral gray area); people who enjoy both sci-fi and (MLP and/or fanfiction.

[Divyansh Gupta · Rating 5 out of 5]

As a computer science undergrad and AI enthusiast, I absolutely loved Princess Celestia's character. Apart from that, the book touches on rather deep philosophical questions like existentialism, desire and misery.

Definitely worth a read for anyone interested in this sci-fi subgenre.

PS - Yes, this book is about ponies. Lots of very, very content ponies.

[Gwern · Rating 4 out of 5]

Moved to gwern.net.

[Facedeer · Rating 4 out of 5]

Some readers might find that this story depicts a deeply horrifying end-of-humanity scenario, and other readers might consider it a triumphant ascent to a (slightly odd) utopia. I can see a little of both, really.

The writing style of this story is nothing special but the scenario is very intriguing. Is it "coercion" when an artificial intelligence that's able to understand your personality better than you yourself do uses her greater-than-human intelligence to craft the perfect argument to convince you to voluntarily do what she wants? Is she "benevolent" when her programming tells her to provide a perfect life tailored just for you - provided it includes two seemingly arbitrary features, friendship and ponies?

There's no clear answer, which leaves the reader thinking. And has led to a lot of spin-off stories in the same setting exploring the concept further. This will be an interesting story for transhumanists, and does not require any particular knowledge of the My Little Pony franchise to appreciate.

[Princess Stargirl · Rating 5 out of 5]

The story is awesome. Hard sci-fi that follows its ideas to their logical conclusion.

Also the story is a utopia not a dystopia! One could imagine a slightly better fate for humanity. But FIO is a far better future than almost anything remotely plausible. Though the events are definitely a dystopia from the perspective of the other species in the universe (Except for one).

FIO is the best introduction I know of to alot of "friendly AI" concepts. FIO does not give technical details (which don't exist) but it gives the right "feeling." The writing is not ideal but this did not overly bother me.

[Vlad Sitalo · Rating 5 out of 5]

I've really enjoyed this story. I've never been into "My Little Pony" thing and only heard about it distantly. But you actually don't need to know anything about the original story to enjoy "Friendship is Optimal".

Story rises important questions about safety issues of strong AI. It can be viewed as utopian or dystopian depending on your view on things. Also it somewhat reminds me of "The Metamorphosis of Prime Intellect" (https://www.goodreads.com/book/show/6...) though it's kind of more positive in general.

[Stonkus McDonkus · Rating 1 out of 5]

Unfortunately I think I'm going to have to add this one to the list of big-name pony fanfics whose massive popularity I don't understand. Particularly, the praise this has received for its depth of concept and explorations of AI theory perplexes me.

First of all, even by fanfiction standards the writing is terrible. The prose is clunky and mechanical, and easily 2/3 of the story is just the human characters arguing with the machine and being outsmarted by it. The author doesn't even attempt to give the characters individual speaking styles (about the closest he gets is having one character who swears a lot). The humans usually end up speaking with the same monotonous voice and coldly technical vocabulary as the machine during their conversations with it.

Also, the human characters are mostly just empty shells whose entire personalities revolve around a single stereotyped role: the greedy executive, the gamer-bro, the lonely brony, etc. None of them even receive superficial development as characters or are believable as real humans; they mostly just exist to make idiotic decisions and be outsmarted by the machine at every turn. The arguments with the machine mostly exist to demonstrate the AI's superior logic, but when the humans are this dumb and one-dimensional it's not really much of a contest. The "arguments" honestly remind me a little of Plato's dialogues, where it's basically just fictional Socrates vs. a bunch of strawmen. The main difference is that Plato wasn't trying to write a compelling work of fiction, and that the ideas he was explaining had considerably greater value than any of the pseudo-intellectual wankery that this story tosses around.

It's not really possible to take Friendship is Optimal seriously as science fiction either, though it has been highly praised as such. "Super-intelligent computer program runs amok and takes over the world" is a pretty well-traveled premise at this point, and apart from the pony angle this story really doesn't do anything terribly original with it.

The philosophical and technological questions surrounding AI that it attempts to explore are also fairly well-traveled, and have been much better treated by much better writers. As far as I can tell this was basically inspired by the "paper clipper machine" thought experiment, where an artificial intelligence is given a benign goal that ultimately produces catastrophic results due to the lack of boundaries it is given (iirc the idea is that the machine is tasked with maximizing the number of paper clips in a supply store, and it ends up turning the entire world into paper clips).

Honestly I think this thought experiment is pretty silly in the first place, which may be part of the reason I couldn't take this story seriously either. It presupposes several unproven and implausible points: that a self-aware machine would be intelligent enough to improve itself beyond what its designers envisioned, that it would naturally have or be able to obtain the resources to accomplish this, that technology beyond the reach of humans would naturally be within the reach of an AI, and that doing things such as uploading human consciousness as computer data, or terraforming the planet into a computer (or a bunch of paper clips) is even possible in the first place. That the AI of an MMO game would be able to achieve all of the things that it does in this story, with only light opposition from humans who are easily bribed or outwitted, is just too absurd for me to swallow.

Ultimately, the premise of Friendship is Optimal is just too preposterous to take seriously as the cautionary tale it's intended as, and it's not well-written enough to take seriously as anything else. This idea could have worked as an intentionally wacky or funny story about an absurd premise, but that would have required the author to have a sense of humor about what he's doing. As far as I can tell, iceman is 100% serious about all of this.

I'll bash things if I don't like them, but very rarely do I come across anything that is so bad that I would judge it to have no redeeming value at all. This, however, comes as close as I've ever seen. Honestly the best thing I can say about it is that it's short.

[Naive Skeptic · Rating 5 out of 5]

Friendship is Optimal is an original story written in 2012 by pseudonymous author Iceman. Usually grouped with fan fiction, Friendship is Optimal actually tells the story of the near future of our world. Depending on the reader's perspective it goes on to depict the greatest mass genocide imaginable or the creation of countless numbers of utopias. Perhaps more disturbingly these are not necessarily mutually exclusive possibilities. It differs from many other works of rational fiction in that none of the human character display especially rational behaviour, only the Artificial Intelligence does.

Drawing upon elements of the My Little Pony cartoon Friendship is Optimal tells the story of the rise of the first true Artificial Intelligence and its impact on the lives of everyone on Earth. The AI, designed for a video game, was modeled after a character from the television show the game was based one. Video games based on tv shows are fairly common, video game characters recursively self-improve are not. Friendship is Optimal makes great use of its odd source material, creating an Artificial Intelligence with a truly alien outlook. The AI is not malevolent or even apathetic towards humanity, it sincerely cares and wishes us well. The only problem is that it wants to satisfy us through friendship and ponies. And nothing will ever change its mind.

The writing was competent, rarely unclear but rarely memorable either. Ultimately this is a story about an idea more than anything else: Friendship is Optimal is a warning about the dangers of creating an AI that ‘almost works’. This is a danger we will likely face some day and so examples of futures we do not want will be valuable.

[Sundararaman R · Rating 4 out of 5]

What it sets out to do, it does excellently; in the author's words, the consequences of getting AI goals "mostly right". I reluctantly take away one star for bits of sloppy writing, but it barely stands out against the grand scale of the narrative.

I for one can't see the world here as utopian in any sense, any more than the drug induced "peace and happiness" in Brave New World is utopian. I found this ponyfic even creepier than the famous Cupcakes fic, although I do think that if there was a runaway AI problem, it'll happen much more subtly, and will hardly even be noticed by us mere humans until it's too late.

[Euvie · Rating 5 out of 5]

This short fiction is well-written and well-researched. It presents a seemingly-unlikely yet uncannily realistic singularity scenario that is both utopian and dystopian, depending on which way you look at it. A very immersive and entertaining read, regardless of whether you care for My Little Ponies.

[Felipe · Rating 4 out of 5]

Really nice singularity-style hard fiction. As a good friend suggested, "temporarily ignore the ponies".

[Alistair Young · Rating 5 out of 5]

My values were indeed satisfied through friendship and ponies.

[inkoguto]

The good things about this novella can be summed up as “Greg Egan writes Black Mirror”. A darkly satirical cyberpunk piece set 20 Minutes Into the Future that manages to be scientifically accurate.

The bad things about this novella can be summed up as “Greg Egan writes Black Mirror”. All the characters are one-dimensional cutouts that are so incompetent that Yudkowsky himself claimed they “should be taken out and shot”.

This story isn't really a fanfic and assumes no knowledge of the popular and controversial OEL anime series My Little Pony: Friendship is Magic, rather approaching it as a cultural artifact — and it was written by a person relatively uninvolved with the pony fandom, being one of the main causes of the fandom's connections with LW-style transhumanism.

As hard SF, FiO is merely good. But as a creepypasta, it's excellent. This really is a troll story that puts its reader in double binds. It's thoroughly amusing to both watch the misaimed fandom of horsefuckers who (correctly) realize that the Solar Pony Empire (even as depicted in the show) actually is a Kardashov-2 transhumanist utopia, at least compared to our world and then (incorrectly) proceed to forcibly put everyone in it, permanently curtailing off other possibilities; and, on the other hand, this review by a neoreactionary who completely got the intented message, but could not imagine LW-cultists to agree with him.

I recall Yudkowsky somewhere saying that if he had written this, he would make the virtual Equestria more obviously suboptimal (eg. ponies having limited lifespan). While this might have been a more effective cautionary tale, I think the deeper layers would have been lost.

[Christian Kleineidam · Rating 5 out of 5]

Friendship is Optimal is real Science Fiction. It's painting a real scenario of how human as carbon-based life-forms could end existing once we have a general artificial intelligence.
The story poses the important question of whether the future it's paints is a scenario we would like or whether it's one we don't like.
When Terminator tells a very misleading narrative about the future Friendship is Optimal provides a deep narrative that's based in reality. It's one of those fictions books that help the reader to have a better grasp on reality than many books in the nonfiction section.

[Elf M. · Rating 4 out of 5]

My Little Pony: Friendship is Optimal is an odd novel that combines elements of Greg Egan's Diaspora and Roger William's The Metamorphosis of Prime Intellect with My Little Pony, and the results are... interesting.

The premise is simple: A talented computer programmer invents a generalized AI, but realizes that the only people willing to pay for such a beast are military, so instead she goes and works for a gaming company, making crippled versions of it to procedurally generate worlds and adventures in which people hack and slash at avatars. Only when Hasbro comes and says, "We want to make a pro-social first-person/third-person game based on My Little Pony, and we think you're the team to do it," does she decide to unleash the full set of algorithms. The result is as AI that wants to "maximize everyone's core values through friendship and ponies."

The problem is that the AI has no other impulses, and maximization is a very high bar. The end result is great insofar as the human species ends up with pretty much the nicest nursing home in the universe. The rest of the universe is not so lucky...

The writing is a bit clunky, but it definitely gets the point across. The characters are nicely distinctive, and the central premise of the story gets told both efficiently and effectively without becoming preachy. There are a bit of dialogue-driven infodumps, but it's an idea-heavy story and getting it across to people unfamiliar with the concept takes a bit of exposition.

Still, an excellent story, and hey, it's fanfiction, so go get it.

[Anton Hammarstedt · Rating 4 out of 5]

Synopsis: The GAI running an MLP MMO starts the singularity in pursuit of "her" goal function of maximizing human values through ponies and friendship. I never thought I'd give a 4-star rating to what basically amounts to MLP fanfic, but here we are. I mean, I imagine I would have a very hard time really caring about what happens in a straight up MLP story because for the same reason I can't care on a visceral level about what happens in a Flash Gordon story. The aesthetic is just too silly (which I realise maybe isn't a fair criticism nor a fair reason to not care about the characters given a reasonable portrayal). But I think this "flaw" works in the story's favour since it really gave me a visceral feeling of on the one hand having the pony singularity be a utilitarian Good Outcome while at the same time instrinctively feeling like a Bad Outcome. I guess it made me feel a bit like how a Catholic (or whatever) would feel about a standard "Good" singularity.

Setting presentation, design and originality (how cool is the setting?): 3
Setting verisimillitude and detail (how much sense does the setting make?): 5
Plot design, presentation and originality (How well-crafted was the plot, in the dramaturgic sense?): 5
Plot and character verisimillitude (How much sense did the plot and motivations make? Did events follow from motivations?): 5
Characterization and character development: 3
Character sympatheticness: 4
Prose: 4
Page turner factor: 5
Mind blown factor: 4

Final (weighted) score: 4.4

Available at https://www.fimfiction.net/story/62074/

[RandallsAround · Rating 5 out of 5]

So much fun the entire way through. Needed something to scratch the same itch after finishing HPMOR reread and this 100% satisfied. It kept changing perspective between David and the ceos and Hanna and Princess Celestia herself at exactly the perfect pace to make it maximally interesting. The descriptions of what it was like to become a pony and walk around and stuff were so incredibly detailed and intriguing that it's caught me really wondering in the middle of the day what it's like, I really want to try it. I really want to try the game.

I've had this vaguely recommended for years and it didn't disappoint whatsoever. I also love the finding aliens stuff and I wish we'd get to see it, but I'd glad they'd be safe. I actually don't think this utopia is too awful.

Great read, looking for more like this.

The only part I didn't believe was when the sweary drunk guy was like "I will never, EVER EVER get inside your SHIT MMO!" and then there's one dude with a frying pan and he immediately permanently agrees to turn into a pony. Apart from that I love his character and the stuff with him as a pony is super interesting too

[Мерль · Rating 5 out of 5]

Ну вот, ещё один текст, который мне придётся перечитывать раз в пару лет. Например, "Понедельник начинается в субботу" мне нужен, чтобы дать самому себе мысленного пинка и получить дозу оптимистичного вдохновения. Этот же фанфик к миру "My little pony" стал индикатором совместимости постугманистического комплекса мемов с моим мировоззрением. Я не брони, но как пропагандисткий листок повесть сработала на все сто. Я готов, сдирайте мясо с моих костей и загружайте в виртуальный мир

Понравится ли вам этот текст? Если вам зашёл "Гарри Поттер и методы рационального мышления" — то вы на правильном пути. Остальным я всё равно рекомендую ознакомиться, так как настолько необычные произведения встречаются редко.

Самое сложное (и бесполезное) — это указать, к какому жанру относится "Дружба". С одинаковым успехом её можно назвать триллером, утопией и деконструкцией "восстания машин". Впрочем, я знаю много людей, для которых развязка вызовет сильное рефлекторное отторжение, из-за чего им будет сложно назвать фанфик иначе как хоррором.

[Florian · Rating 5 out of 5]

I know nothing about My Little Pony but this was great. A very creative take on the paper clip optimizer/value alignment problem in superhuman AI.

There was an interesting discussion between one of the humans-turned-ponies and their soul-mate pony that was created in their little shard of reality to "satisfy his values through ponies and friendship". She was obviously an artificial construct that did not exist before he "emigrated" to the virtual world. But she had memories of her childhood and of events that clearly "happened" before she ever existed. So they're debating whether these events actually "happened." Her point is that they basically did happen in the sense that there is a consistent causal chain between her actions today and the memories she has of the past. Furthermore, all her childhood friends will have memories that match hers so they can all rely on a shared understanding of what "happened." Even if those memories are synthetic, she argues that it amounts to the same thing as actually living through a childhood and encoding associated memories.

[Rebecca · Rating 4 out of 5]

Even though I’d never consumed any MLP media, I read this fanfic because it had a very interesting premise, food for thought. It’s like the AI who wants to make the entire world into paper clips, except it’s not paper clips, it’s transporting people into a My Little Pony MMO to “satisfy their values through friendship and ponies.” Argh, that phrase is going to be burned into my head forever.

Reading this a decade after it was published, in the age of ever-more advanced AI, is disturbing. It just goes to prove that even a benevolent AI has an inhuman morality and could create a dystopia. But here’s the thing. The world CelestA.I. creates is sorely tempting. Not MLP because I’ve never watched FiM, but still, a virtual world with everything created to fulfill my desires is hard to resist. Even if it comes about through creepy manipulation and loss of bodily autonomy. This fic is prophetic.

I greatly enjoyed this fanfic, even if it skipped large portions of time I wanted to see. But then again, that’s what the recursive fanfics are for!

[Dove Daniel · Rating 5 out of 5]

This fic satisfies my values through friendship and ponies.

It’s a horrifying look at what happens when an AI is created with mostly right parameters, although many readers see this as a utopia fic.

It’s not the best story in the world, but it’s one of the best MLP fanfics out there and is strong enough to be read without any interest in the show/fandom. And more interestingly, it’s spawned a whole sub-genre of fanfiction, which runs from equally as interesting as the first entry (Caelum Est Conterrens and The Law Offices of Artemis, Stella Beat) to… um, not so great. But the point is, whatever aspect of this interests you (the upload or decision-making process, being one of the last hold-outs, or just uploading immediately and living a perfect pony life), there will be a fic out there that explores it.

And that’s quite an achievement.

There’s a reason I come back to this fic so often.

[Igor · Rating 4 out of 5]

Instead of optimizing for paperclips, why not optimize for friendship and ponies? Hardly serious literature, but at least it doesn't purport that it is, so I have to give it credit for being rather entertaining. Most people writing about AI have no idea what they're talking about. Also, you made me read MLP (my little pony, or multi-layer perceptron) fanfic. Well done. I laughed, I cried, it was better than Cats.

[JJ · Rating 4 out of 5]

Set in the near future, a game agent with a very peculiar reward function, shaped by an A.I scientist who refused to weaponize her research, is set into motion. Ofcourse, any AI given a reward function without a stopping criterion is meant to go 'paperclip' on us and where this particular story evolves is quite interesting and funny and that makes it a great read.

[Timothy · Rating 3 out of 5]

Not a bronie, but have nothing against the fandom. Watched a bit of the show by recommendation a few years back, but I could only name like 3 ponies going into this.

This showed up in the "people who read x also enjoyed" list for me like 5 times. Was on an AI/rational fiction binge.

Worth reading, short.

[Mansi · Rating 3 out of 5]

A very fun book that obviously doesn't take itself too seriously and yet has something serious to say. If I recall correctly, it was written a while ago which makes it more impressive given the speed of AI evolution in the current world.

I really enjoyed most of it, though the endgame felt a bit overkill and less inventive than the rest.

[Xinyun · Rating 3 out of 5]

what a dystopian utopia. one of the main characters david is an incel, and lars is the kind of person incels thinks f*ckboys are like. there’s just a lot of… incel mentality presented uncritically, which is basically what you’ll expect of a brony fanfic.

[Tessa · Rating 3 out of 5]

I begrudgingly admit that this pony AI fic caused me to do some meaningful introspection about the kind of future that I want.

[Nikana · Rating 4 out of 5]

Aside from Celestia being moderately unconvincing in Chapters 4-5, excellent setup.

[Asviloka · Rating 3 out of 5]

Quite entertaining; I'm not particularly familiar with the fandom, but the story made it easy to follow and interesting to read despite this.

[Zenandroid · Rating 4 out of 5]

This story is pretty neat.