Inference Engineering

by Philip Kiely

Inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, more reliable serving of generative AI models in production. While the potential and impact of inference are becoming clear, the space is young. There are relatively few people working on inference, and newcomers can become experts quickly. There are opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

Inference Engineering is your guide to becoming an expert in inference. It contains everything that I’ve learned in four years of working at Baseten. This book is based on the hundreds of thousands of words of documentation, blogs, and talks I've written on inference; interviews with dozens of experts from our engineering team; and countless conversations with customers and builders around the world.

Genres: Technical, Artificial Intelligence, Engineering, Technology, Software

260 pages, Paperback

First published April 7, 2026

Reviews

[Malathi Sankar · Rating 5 out of 5]

A thoughtful and practical introduction to inference engineering.
This is an important read not only for engineers building AI systems, but also for engineering and technology leaders. Inference engineering for generative AI is fundamentally different from traditional ML inference. The book spans models, hardware, software, runtime optimization, and production systems, with constant tradeoffs between latency, cost, reliability, and quality. Philip Kiely explains these concepts with clarity and depth, making a complex topic approachable without oversimplifying it. I learned a great deal and gained a much deeper appreciation for what it takes to build reliable, scalable AI products. Highly recommended.

[Sourav Tripathy · Rating 3 out of 5]

A good book that introduces you to all the moving parts involved in the serving and inference of LLMs along with video, audio, image models. Also beautifully makes you aware about tradeoffs involved between the current optimizations techniques like speculative decoding or quantization or caching.

[Arun · Rating 3 out of 5]

3.5 (Pdf version although I bought a dead tree version)

Gives a good overview of the topics for an inference engineer but the meat is in the appendix which has some very good suggestion of books and articles for detailed study.

The writing is tight and without much fluff but I am disappointed by useless uml-like diagrams and not much detail in the main contents.

[Stefan · Rating 3.75 out of 5]

A great signpost to quickly build vocabulary and find your way around the domain. It outlines what’s what, providing a solid foundational understanding of how and why it all matters. It doesn't go too deep; just deep enough for a non-practitioner. A good read before diving into "AI Systems Performance Engineering".

[Mohit Acharya · Rating 5 out of 5]

really cool bird's eye view of the whole inference scene! it lays out the map really well.

[George Veneel Dogga · Rating 5 out of 5]

banger

[Sero c · Rating 5 out of 5]

I work on AI inference. This book is an incredible high level overview of how to run models efficiently and effectively

I recommend it to anyone who is interested in self hosting or working in AI this skill is greatly in demand right now and will give you a long and prosperous career