Deep Reinforcement Learning Hands-On: Apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more

by Maxim Lapan

This practical guide will teach you how deep learning (DL) can be used to solve complex real-world problems.

Key Features Explore deep reinforcement learning (RL), from the first principles to the latest algorithms Evaluate high-profile RL methods, including value iteration, deep Q-networks, policy gradients, TRPO, PPO, DDPG, D4PG, evolution strategies and genetic algorithms Keep up with the very latest industry developments, including AI-driven chatbots Book Description

Recent developments in reinforcement learning (RL), combined with deep learning (DL), have seen unprecedented progress made towards training agents to solve complex problems in a human-like way. Google's use of algorithms to play and defeat the well-known Atari arcade games has propelled the field to prominence, and researchers are generating new ideas at a rapid pace.

Deep Reinforcement Learning Hands-On is a comprehensive guide to the very latest DL tools and their limitations. You will evaluate methods including Cross-entropy and policy gradients, before applying them to real-world environments. Take on both the Atari set of virtual games and family favorites such as Connect4. The book provides an introduction to the basics of RL, giving you the know-how to code intelligent learning agents to take on a formidable array of practical tasks. Discover how to implement Q-learning on 'grid world' environments, teach your agent to buy and trade stocks, and find out how natural language models are driving the boom in chatbots.

What you will learn Understand the DL context of RL and implement complex DL models Learn the foundation of RL: Markov decision processes Evaluate RL methods including Cross-entropy, DQN, Actor-Critic, TRPO, PPO, DDPG, D4PG and others Discover how to deal with discrete and continuous action spaces in various environments Defeat Atari arcade games using the value iteration method Create your own OpenAI Gym environment to train a stock trading agent Teach your agent to play Connect4 using AlphaGo Zero Explore the very latest deep RL research on topics including AI-driven chatbots Who This Book Is For

Some fluency in Python is assumed. Basic deep learning (DL) approaches should be familiar to readers and some practical experience in DL will be helpful. This book is an introduction to deep reinforcement learning (RL) and requires no background in RL.

Table of Contents What is Reinforcement Learning? OpenAI Gym Deep Learning with PyTorch The Cross-Entropy Method Tabular Learning and the Bellman Equation Deep Q-Networks DQN Extensions Stocks Trading Using RL Policy Gradients – An Alternative The Actor-Critic Method Asynchronous Advantage Actor-Critic Chatbots Training with RL Web Navigation Continuous Action Space Trust Regions – TRPO, PPO, and ACKTR Black-Box Optimization in RL Beyond Model-Free – Imagination AlphaGo Zero

Genres: Computer Science, Technology, Artificial Intelligence, Programming, Science, Research, Textbooks

546 pages, Paperback

Published June 21, 2018

Reviews

[Isen · Rating 1 out of 5]

The book covers various topics in reinforcement learning, starting with basic concepts and building the way up. The focus is on application rather than theory, and each chapter presents a worked example with code available on a supporting Github. At first glance this seems like a wonderful resource to learn RL, but the shortcomings soon become apparent.

The first is that the book takes a draw-two-circles-then-draw-an-owl approach with a steep jump in complexity in Chapter 6, when you go from solving a toy problem to solving Pong. This is a problem because to solve Pong you have to deal with a whole host of details irrelevant to machine learning, such as image recognition and the specifics of Atari games (such as pressing fire when the game starts), and with a model that takes at least two hours to train you don't have a lot of leisure to experiment with parameters and architecture.

This leads to another issue, is that you're forced to do this experimentation yourself. Every chapter presents a more or less tidy story of an example that works, with no questions or exercises to work through, so you never really understand why it just works if you just work through the book. You have to stop and experiment yourself after everything you learn.

This sounds like a good thing, but that takes us to my main problem with the book, is that when you start experimenting with the techniques given, you find that they don't "just work" for any problem that differs slightly from what is given in the text. Perhaps this means that for any given task in RL messing around with parameters is more important than the actual model used, but if so, the book should really dedicate more time to just that. What's worse, upon closer examination, even the examples given in the book don't "just work". The DQN solution for Frozen Lake in Chapter 6 does not actually converge, it only seems to because the author for some odd reason decided to evaluate its performance by the best case performance rather than average case. Eventually the model just gets lucky, and the author marks the problem as solved, while the actual performance fluctuates wildly for as many epochs as I cared to run the model for, and fiddling with the parameters didn't help much either. I didn't bother exploring the more complicated models because whereas it is feasible to rerun Frozen Lake until I get something that works, it really isn't with something that takes hours to train.

Is there some use you can get from this book? Yeah, probably. Are there better options? Sure, just about any tutorial.

[ShawnLeeZX · Rating 4 out of 5]

I found this book a good complement to Reinforcement Learning by Richard S. S. and Andrew G. G.. It saves me time gleaning over papers to know the implementation details of the papers. It also provides a simplified thread of the development RL methods without possibly the biases in the more theoretical book: in the sense that I want to figure out why the algorithm works and what their limitations are without being first biased by the opinions of the authors; this is desirable because many of the explanations of why the algorithms work in machine learning are misleading due to inadequate understanding of the practitioners in the field.

[David · Rating 1 out of 5]

Not the best presentation and the applications should be cut down for simpler reading. Code examples are not the best because of this.

Not enough focus on continuous observation and action spaces.

Not enough focus on custom envs using simple simulations of processes. That should be the starting point not prebuilt ones based on games etc. It will simplify the exposition and make this more useful to people as a book.

[René Pázman · Rating 4 out of 5]

Good introduction to Deep RL for people interesting in understanding algorithms in detail. Suitable for learning by example, not for learning strong theory. For some examples, there was a need to download full code in order to understand them well (context of the code in the book was sometimes missing).

[Mehdi · Rating 5 out of 5]

Must read for RL practitioners

[Chien Chin-yu · Rating 5 out of 5]

寫的蠻好的，翻譯也很適當，沒有任何文字上的閱讀障礙。每一章也都有程式碼範例，解釋程式碼的部分也不會太過瑣碎，幾乎都是重點部分。

缺點（？）大概是沒啥廢話，資訊的濃縮密度超高，加上內容不簡單，導致讀起來很痛苦😖，一半以後就以看懂原理為主，之後再來詳讀程式碼。

[Ramasubramaniam · Rating 5 out of 5]

A very practical book.

[Tom · Rating 2 out of 5]

This book should have been a website.
The author intermixes code snippets with explanations.
Many code snippets are of the type "nothing changes here".
The author goes from the top of a file down and explains every line rather, than focusing on
changes or objects