Data-oriented design: software engineering for limited resources and short schedules

by Richard Fabian

The projects tackled by the software development industry have grown in scale and complexity. Costs are increasing along with the number of developers. Power bills for distributed projects have reached the point where optimisations pay literal dividends. Over the last 10 years, a software development movement has gained traction, a movement founded in games development. The limited resources and complexity of the software and hardware needed to ship modern game titles demanded a different approach.

Data-oriented design is inspired by high-performance computing techniques, database design, and functional programming values. It provides a practical methodology that reduces complexity while improving performance of both your development team and your product. Understand the goal, understand the data, understand the hardware, develop the solution.

This book presents foundations and principles helping to build a deeper understanding of data-oriented design. It provides instruction on the thought processes involved when considering data as the primary detail of any project.

Genres: Programming, Computer Science, Nonfiction, Technical, Software, Technology, Science

Kindle Edition

Published October 7, 2018

Reviews

[Iman Jahandideh · Rating 1 out of 5]

As an OO game programmer I was interested about the Data Oriented approach. But the first question that I needed to be answered was "Why?" and in my opinion the book failed to answer this question. I agree DO is more performant than OO but for other claims the author either didn't tried to give any explanations or his explanations and examples was very weak.

A great part of the book was also devoted to "How". The main thing that peaked my interest was using databases for game. But maybe since I didn't understand the "Why?" part, most of the approaches seem too short and too vague to give any new insights to the DOD.

[Aaron · Rating 3 out of 5]

There are a lot of good ideas in here, but they are not terribly well explained, especially to those outside of the game industry. This topic is clearly relevant to a wider audience, but he is so focused on game developers that he fails to make the material very accessible to anyone else. More details and better examples would go a long way. Perhaps if the author spent less time attacking OO he would have had more energy to spend on the topic at hand.

[Stijn · Rating 2 out of 5]

Not that this is a bad book, but if you’re not a game developer writing c++ counting your cache hits, this book isn’t for you.

[Adelbert · Rating 5 out of 5]

Very well written and codifies data-oriented design practices into an organized book. If you enjoyed Mike Acton's CppCon'14 (https://youtu.be/rX0ItVEVjHc) you will enjoy this book. Even if you are not a game or graphics dev, it is worth reading if only to get a glimpse into an infrequently discussed perspective on software engineering - hardware-aware programming - applicable no matter where in the proverbial stack you are. To paraphrase Professor Alan Perlis, "a [philosophy] that doesn't affect the way you think about programming, is not worth knowing," and this book has definitely changed the way I think about programming.

[Ruben · Rating 5 out of 5]

Honestly the best and most necessary programming book I've read in recent memory. I'd recommend it to any kind of programmer, but if you're into systems programming, embedded or game development I'd qualify it as a must-read.

Does a great job at making you take a step back from practices that have established themselves in programming over the past few decades that we still blindly follow to our detriment. Justifies its claims with very well written arguments and shows a different path that we should take when we sit down to solve our problems with code.

[Güçhan · Rating 3 out of 5]

I picked up this book mainly because I was working with ECS (Entity Component System) at work. While it briefly touches on ECS, it goes far beyond that, diving into data at a much more fundamental and lower level, which I found useful.
Like most computer science textbooks, this one requires active studying—you can’t just read it like a novel. It’s also a highly relevant read in today’s era of concurrency, GPU utilization, AI, and crypto, I think.
The book includes many references to hardware, which inspired me to explore and learn more about that topic as well.

[John Roodt · Rating 5 out of 5]

Understanding Unity’s move to DOTS & ECS

I came across this book whilst browsing for some insight into Unity’s switch to DOTS (the Data Oriented Technology Stack) which includes ECS (the Entity Component System). I certainly have a far better idea as to what motivated them to make the switch, & now confident that this is the direction I should take too

[Trung · Rating 3 out of 5]

The book is rightfully good, but it's about 3 times longer than it needs to be because it regurgitates its ideas over and over again far too often. The main idea the book is trying to sale is to restructure your video games around what is called these days an Entity Component System design from an Object Oriented design. The motivation to move over are:
1. OO design makes it difficult to tackle composition of in-game structures as the project grows.
2. OO design is not cache-line friendly. Structures of vector is the suggested solution.
3. OO design imposes additional control flow complexity in your code, through virtual function dispatches and operations over not-normalized data structures (i.e. nullable)
I think there are definitely better sources out there to learn about ECS in practice; even the source code of an ECS system is probably far more enlightening. For the motivation, one should look up how memory works. The book doesn't go sufficiently deep into either topic so I can't sincerely recommend it. It seems fit for people who already have an OO game engine running and needs some convincing to take the leap of faith over to ECS.

[Pablo Weremczuk · Rating 3 out of 5]

The book gives a good idea about what's data-oriented design is about.

Still, falls short on giving good examples. As an example, there is an example regarding "false sharing", which is when you have two threads and they invalidate the cache of the other writing in close-enough memory addresses. I found the example extremely convoluted and hard to follow, compared with other examples in a stack overflow. Using plain C makes more sense than using C++ for examples.

Still is the only book you have to explore this design philosophy, and is fairly good.

[Paweł Rusin · Rating 4 out of 5]

There are some programming paradigms I've learned over years that changed the way I code, and DOD is one of them. This is probably most well-known book on a subject and is a great introduction to the subject. Like with a lot of DOD, most examples are from game programming, so sometimes it's hard to map it to other domains in programming. Also, I wish it had more code samples to introduce and explain the concepts further.

[Ivane · Rating 5 out of 5]

Passion motivated and well written book about software design paradigm that was was born [or reincarnated] within field it serves [ever-changing game design and development]. Paradigm which [performance benefits aside] embraces change--be it design or implementation structure--and assumes knowing of [at least] something about actual problem of data transformation on an actual physical hardware.

[molty cheese · Rating 2 out of 5]

Confusing

The author likes to ramble on presented topics and sometimes segues into completely unrelated themes and not finishes his previous thoughts. Because of this the book reads more like a thought dump rather than a structured source of knowledge.
While there are nice ideas in this book, I don't think it is worth deciphering them from the incoherent wall of text.

[Roman Kashitsyn · Rating 3 out of 5]

I really wanted to like this book. It has some very good ideas, like the connection between normalisation in relational databases and ECS. However, the language is rather vague and hard to follow. The book could be twice as short without losing anything of importance.
I share the hope of the author that more books on the topic will follow.

[Isaac Serrano · Rating 4 out of 5]

This is a paradigm shift book. If you program you should read it, full stop.

Everything this book says makes sense. The only reason I don't 5/5 it is that I haven't /tried/ it enough yet, and I feel there's some argument missing somewhere.

[Julian Schrittwieser · Rating 4 out of 5]

Interesting view on data oriented design, but sometimes more subjective opinion than objective coverage of available methods. Also somewhat restricted to games, other domains quite different requirements.

[Victor Debone · Rating 5 out of 5]

This book contains unusual density of information (high), and surprised me with an unapologetic style of getting to the core of things, not trying to getting you up to speed with long introductions or trying to appeal to a larger public. I wish technical books were all like this.

[Simon Eskildsen · Rating 2 out of 5]

Didn't feel I got much out of this. Andrew Kelly's
talk (https://media.handmade-seattle.com/pr...) and
the OG Mike Acton talk (https://www.youtube.com/watch?v=rX0It...). Instead
of reading this book, read those two and then program something.

[Ryan Riley · Rating 5 out of 5]

Highly recommended for anyone working with data.

[Andrew]

I enjoyed this work about high performance computing.

[Matt Davis · Rating 4 out of 5]

ngl skipped over sorting

[Rasmus Källqvist · Rating 3 out of 5]

At the time of writing this is one of the few books on the topic, and it's an interesting one at that. We're introduced to the idea of viewing the programs we're writing foremost as a sequence of data transformations, i.e. a sequence of functions that take data and produces new data, and as such can design our systems not by modelling the domain in code but in looking at what data we need as input to produce the desired output, with as little transformations needed in-between.

I think the book does an excellent job at introducing this way of viewing software design, and the technique of using relational databases to build up a data oriented correspondence to object relations is great.

My complaints are that there could be some more formalisms that more clearly outlines what data oriented design is actually supposed to be, rather than through examples, and to reduce the unneeded jabs at object oriented programming, and the overly focus on game programming.

All in all, it's a worthwhile read, albeit feeling sometimes more like a handbook than an introduction to a paradigm. Pick it up!

[Mikhail Filatov · Rating 1 out of 5]

This book has very little to do with design and nothing with "limited resources and short schedules". It's a set of incoherent rumblings against OOP. The only (probably) relevant one is about performance overhead of using virtual functions.
The author reveals to us in the second chapter a hidden gem -relational databases :)
Really, he describes them with enthusiasm of Larry Ellison selling Oracle "2.0". Maybe SQL DBs are not used in game development, but otherwise they (and OO languages) are pretty much mainstream for the last 25+ years.
There are some chapters about perf. optimizations in game development, but as I'm not a game developer it's difficult for me to evaluate - and nothing in the title and subtitle of this book hints that it's for game developers only.