Giao Phan’s Reviews > Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing > Status Update

16. Scaling experiment analysis
Data processing: sort and group; clean; enrich
Data computation
Summarize and visualize

15. Ramping experiment exposure
Here this chapter mainly recommends the gradual exposure of treatments to the users, not 0 to maximum power ramp (50%) but by smaller rings of the experimental units

13, instrumentation
Again some caveats and then to emphasize that instrumentation is necessary

12, client side experiments
Here the focus is the difference of experiments on thin vs thick clients, and the caveats.
It would be interesting for those who are engineering these experiments. Feel like it is more of a check list.

Chap 11: observational causal study
The methods in this umbrella are described as alternative methods to controlled experiment, to be applied in cases where controlled experiments are not possible. However, there are a plenty of examples where the results of observational causal study are far wrong from controlled experiment.
Honestly i cannot understand yet each methods in this chapter

10. Other complementary methods
This chapter lists other methods than experiment and their pros and cons. It creates some mental mappings in my heads when to use which methods, and many times we need to combine different methods to achieve the goal.
Controlled experiment maybe a demanding task to small companies without supporting platform.

Chap 8: institutional memory and meta data analysis
- organisation can keep track of the historical experiments they run and does meta analysis to get insights
- plenty of use cases

Chap 7: metrics for experiments and overall evaluation criterion
- experimentation metrics should be measurable, attributable (to features), sensitivity and timely. So not all business metrics can be used as experimentation metrics
- to combine metrics to OEC, we can normalize all metrics then weigh them
- some laws to be cautious with correlation and causal relationship.

Continuing chapter 6.
- how to formulate metrics
- evaluating metrics
- metrics can evolve
- some side notes about guardrail metrics and metrics gamification
So the above contents are more of listing. However i find it fun to read how ingenious humans can be in maximizing metrics by gaming around :))