What do you think?
Rate this book
Testing AI: Engineering Confidence in Non-Deterministic Systems
Previously published as Testing Engineering Confidence in Non-Deterministic Systems by Jason Arbon.
Second edition, reformatted in a compact 7 x 10-inch hardcover.
AI makes software faster to create and harder to trust.
Testing AI is a practical operating manual for shipping AI systems with confidence. It is written for developers, testers, QA leaders, AI builders, product engineers, architects, technical leaders, and executives who need evidence before putting AI-generated code, agents, chatbots, search systems, RAG workflows, or model-backed products into production.
This book moves beyond one-off demos and simple pass/fail testing. It shows how to measure behavior under uncertainty, design useful evals, use human and LLM judges responsibly, reason about sampling and confidence intervals, test generated code and tool-using agents, monitor production behavior, and decide when to ship, hold, canary, shadow, or roll back.
Inside, you will learn how
Evaluate non-deterministic systems across repeated runs, slices, risks, and real user behavior.
Build evals, rubrics, release gates, and production monitors that support actual shipping decisions.
Use statistics such as confidence intervals, p-values, NDCG, precision, recall, and F-scores without fooling yourself.
Test RAG systems, AI agents, prompt injection defenses, generated code, guardrails, cost, latency, and rollback plans.
Work with human raters and LLM judges while accounting for calibration, disagreement, bias, and fallibility.
Understand the emerging discipline of confidence the practical work of deciding whether AI behavior is trustworthy enough to release
Along the way, Jason Arbon shares hard-earned lessons from Microsoft, Bing, Google, Chrome, test.ai, and testers.ai, including production failures, misleading metrics, flaky automation, biased labels, broken rollbacks, and the strange new problems created when AI starts helping test AI.
Testing is the doorway. Confidence engineering is the destination.
Second edition, reformatted in a compact 7 x 10-inch hardcover.
AI makes software faster to create and harder to trust.
Testing AI is a practical operating manual for shipping AI systems with confidence. It is written for developers, testers, QA leaders, AI builders, product engineers, architects, technical leaders, and executives who need evidence before putting AI-generated code, agents, chatbots, search systems, RAG workflows, or model-backed products into production.
This book moves beyond one-off demos and simple pass/fail testing. It shows how to measure behavior under uncertainty, design useful evals, use human and LLM judges responsibly, reason about sampling and confidence intervals, test generated code and tool-using agents, monitor production behavior, and decide when to ship, hold, canary, shadow, or roll back.
Inside, you will learn how
Evaluate non-deterministic systems across repeated runs, slices, risks, and real user behavior.
Build evals, rubrics, release gates, and production monitors that support actual shipping decisions.
Use statistics such as confidence intervals, p-values, NDCG, precision, recall, and F-scores without fooling yourself.
Test RAG systems, AI agents, prompt injection defenses, generated code, guardrails, cost, latency, and rollback plans.
Work with human raters and LLM judges while accounting for calibration, disagreement, bias, and fallibility.
Understand the emerging discipline of confidence the practical work of deciding whether AI behavior is trustworthy enough to release
Along the way, Jason Arbon shares hard-earned lessons from Microsoft, Bing, Google, Chrome, test.ai, and testers.ai, including production failures, misleading metrics, flaky automation, biased labels, broken rollbacks, and the strange new problems created when AI starts helping test AI.
Testing is the doorway. Confidence engineering is the destination.
524 pages, Hardcover
Published October 17, 2026
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
No one has reviewed this book yet.