What do you think?
Rate this book
Experimentation for Engineers: From A/B testing to Bayesian optimization
Optimize the performance of your systems with practical experiments used by engineers in the world’s most competitive industries.
In Experimentation for From A/B testing to Bayesian optimization you will learn how
Design, run, and analyze an A/B test
Break the "feedback loops" caused by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization
Clearly define business metrics used for decision-making
Identify and avoid the common pitfalls of experimentation
Experimentation for From A/B testing to Bayesian optimization is a toolbox of techniques for evaluating new features and fine-tuning parameters. You’ll start with a deep dive into methods like A/B testing, and then graduate to advanced techniques used to measure performance in industries such as finance and social media. Learn how to evaluate the changes you make to your system and ensure that your testing doesn’t undermine revenue or other business metrics. By the time you’re done, you’ll be able to seamlessly deploy experiments in production while avoiding common pitfalls.
About the technology
Does my software really work? Did my changes make things better or worse? Should I trade features for performance? Experimentation is the only way to answer questions like these. This unique book reveals sophisticated experimentation practices developed and proven in the world’s most competitive industries that will help you enhance machine learning systems, software applications, and quantitative trading solutions.
About the book
Experimentation for From A/B testing to Bayesian optimization delivers a toolbox of processes for optimizing software systems. You’ll start by learning the limits of A/B testing, and then graduate to advanced experimentation strategies that take advantage of machine learning and probabilistic methods. The skills you’ll master in this practical guide will help you minimize the costs of experimentation and quickly reveal which approaches and features deliver the best business results.
What's inside
Design, run, and analyze an A/B test
Break the “feedback loops” caused by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization
About the reader
For ML and software engineers looking to extract the most value from their systems. Examples in Python and NumPy.
About the author
David Sweet has worked as a quantitative trader at GETCO and a machine learning engineer at Instagram. He teaches in the AI and Data Science master's programs at Yeshiva University.
Table of Contents
1 Optimizing systems by experiment
2 A/B Evaluating a modification to your system
3 Multi-armed Maximizing business metrics while experimenting
4 Response surface Optimizing continuous parameters
5 Contextual Making targeted decisions
6 Bayesian Automating experimental optimization
7 Managing business metrics
8 Practical considerations
In Experimentation for From A/B testing to Bayesian optimization you will learn how
Design, run, and analyze an A/B test
Break the "feedback loops" caused by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization
Clearly define business metrics used for decision-making
Identify and avoid the common pitfalls of experimentation
Experimentation for From A/B testing to Bayesian optimization is a toolbox of techniques for evaluating new features and fine-tuning parameters. You’ll start with a deep dive into methods like A/B testing, and then graduate to advanced techniques used to measure performance in industries such as finance and social media. Learn how to evaluate the changes you make to your system and ensure that your testing doesn’t undermine revenue or other business metrics. By the time you’re done, you’ll be able to seamlessly deploy experiments in production while avoiding common pitfalls.
About the technology
Does my software really work? Did my changes make things better or worse? Should I trade features for performance? Experimentation is the only way to answer questions like these. This unique book reveals sophisticated experimentation practices developed and proven in the world’s most competitive industries that will help you enhance machine learning systems, software applications, and quantitative trading solutions.
About the book
Experimentation for From A/B testing to Bayesian optimization delivers a toolbox of processes for optimizing software systems. You’ll start by learning the limits of A/B testing, and then graduate to advanced experimentation strategies that take advantage of machine learning and probabilistic methods. The skills you’ll master in this practical guide will help you minimize the costs of experimentation and quickly reveal which approaches and features deliver the best business results.
What's inside
Design, run, and analyze an A/B test
Break the “feedback loops” caused by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization
About the reader
For ML and software engineers looking to extract the most value from their systems. Examples in Python and NumPy.
About the author
David Sweet has worked as a quantitative trader at GETCO and a machine learning engineer at Instagram. He teaches in the AI and Data Science master's programs at Yeshiva University.
Table of Contents
1 Optimizing systems by experiment
2 A/B Evaluating a modification to your system
3 Multi-armed Maximizing business metrics while experimenting
4 Response surface Optimizing continuous parameters
5 Contextual Making targeted decisions
6 Bayesian Automating experimental optimization
7 Managing business metrics
8 Practical considerations
- GenresEngineeringTechnology
247 pages, Kindle Edition
Published March 21, 2023
7 people are currently reading
41 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 2 of 2 reviews
June 25, 2023
I’ve read quite a few books on the subject that stop at the equivalent of Chapter 3. David Sweet doesn’t disappoint as he dives deeper into advanced topics such as Response Surface Methodology, Contextual Bandits, and Bayesian Optimization. Formulas (in moderation), Python and nice plots, make this book an easy read, that reminds me "Bayesian Methods for Hackers". It doesn’t promise so, but this book might not be complete-beginner-friendly. A practitioner will have to do some intelligent work to get the concepts and apply them to their field.
May 12, 2023
Shows the best ways to experiment and evaluate changes in a business. The book is focused on software but can equally be applied across any measurable business metric.
Displaying 1 - 2 of 2 reviews