Status Updates From Reinforcement Learning: An ...

Parameters such as α, n, γ, λ can be picked by trials (n ~ 5 in most cases), instead of grid search, as they seem fairly independent. Methods can be specific to the problem (which maybe a prediction or control, amongst others) with tradeoffs between latency and memory. LLMs help with method selection, discretisation and reduction of state-action space to explore feasibility of ideas.

Had to relearn backgammon.

This subject is rife with higher algebra (especially progressions and probability). Notations can cause mental havoc. Part III seems like interesting prose.

Tabular methods can be understood without programming by creating a scenario with a small state-action space, initialising policy intuitively, and then iterating through the algorithms by setting the variable values. This may not be possible for complex algorithms in the second part - especially off-policy and online search and approximation.

It gets increasingly interesting. Need to revisit a few chapters.

Not as difficult to read now. Going to try non-model based implementations in JAX, but first, remembering my kite equations using plotting libraries.

I had read a different edition of this book for a coursework until a few chapters back, which was available online for free, through none other than Google search. My mind was too shallow to estimate values for bootstrapping, back then. Time to dig.