Inference Engineering

by Philip Kiely

Inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, more reliable serving of generative AI models in production. While the potential and impact of inference are becoming clear, the space is young. There are relatively few people working on inference, and newcomers can become experts quickly. There are opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

Inference Engineering is your guide to becoming an expert in inference. It contains everything that I’ve learned in four years of working at Baseten. This book is based on the hundreds of thousands of words of documentation, blogs, and talks I've written on inference; interviews with dozens of experts from our engineering team; and countless conversations with customers and builders around the world.

Genres: Technical, Artificial Intelligence

260 pages, Paperback

First published April 7, 2026

Reviews

[Malathi Sankar · Rating 5 out of 5]

A thoughtful and practical introduction to inference engineering.
This is an important read not only for engineers building AI systems, but also for engineering and technology leaders. Inference engineering for generative AI is fundamentally different from traditional ML inference. The book spans models, hardware, software, runtime optimization, and production systems, with constant tradeoffs between latency, cost, reliability, and quality. Philip Kiely explains these concepts with clarity and depth, making a complex topic approachable without oversimplifying it. I learned a great deal and gained a much deeper appreciation for what it takes to build reliable, scalable AI products. Highly recommended.

[Mohit Acharya · Rating 5 out of 5]

really cool bird's eye view of the whole inference scene! it lays out the map really well.