Testing AI: Engineering Confidence in Non-Deterministic Systems

by Jason Arbon

AI makes software faster to create and harder to trust.

Testing AI is a practical operating manual for shipping AI systems with confidence. It is written for developers, testers, QA leaders, AI builders, product engineers, architects, technical leaders, and executives who need evidence before putting AI-generated code, agents, chatbots, search systems, RAG workflows, or model-backed products into production.

This book moves beyond one-off demos and simple pass/fail testing. It shows how to measure behavior under uncertainty, design useful evals, use human and LLM judges responsibly, reason about sampling and confidence intervals, test generated code and tool-using agents, monitor production behavior, and decide when to ship, hold, canary, shadow, or roll back.

Inside, you will learn how

Evaluate non-deterministic systems across repeated runs, slices, risks, and real user behavior.Build evals, rubrics, release gates, and production monitors that support actual shipping decisions.Use statistics such as confidence intervals, p-values, NDCG, precision, recall, and F-scores without fooling yourself.Test RAG systems, AI agents, prompt injection defenses, generated code, guardrails, cost, latency, and rollback plans.Work with human raters and LLM judges while accounting for calibration, disagreement, bias, and fallibility.Understand the emerging discipline of confidence the practical work of deciding whether AI behavior is trustworthy enough to releaseAlong the way, Jason Arbon shares hard-earned lessons from Microsoft, Bing, Google, Chrome, test.ai, and testers.ai, including production failures, misleading metrics, flaky automation, biased labels, broken rollbacks, and the strange new problems created when AI starts helping test AI.

Testing is the doorway. Confidence engineering is the destination.

352 pages, Hardcover

Published July 14, 2026

Reviews

[David Levitt · Rating 4.75 out of 5]

I learned a ton from this book. I only wish the author considered integrating his techiques with Gherkin scenarios. When done effectively, the rubric and sample prompts can be introduced in the Given clauses, and the expected respomses can be validated in the Then clauses - all usng the language of the business.

[David Lloyd · Rating 4.75 out of 5]

I learned a ton from this book. I only wish the author considered integrating his techiques with Gherkin scenarios. When done effectively, the rubric and sample prompts can be introduced in the Given clauses, and the expected respomses can be validated in the Then clauses - all usng the language of the business.