Status Updates From Trustworthy Online Controll...

16. Scaling experiment analysis
Data processing: sort and group; clean; enrich
Data computation
Summarize and visualize

15. Ramping experiment exposure
Here this chapter mainly recommends the gradual exposure of treatments to the users, not 0 to maximum power ramp (50%) but by smaller rings of the experimental units

13, instrumentation
Again some caveats and then to emphasize that instrumentation is necessary

12, client side experiments
Here the focus is the difference of experiments on thin vs thick clients, and the caveats.
It would be interesting for those who are engineering these experiments. Feel like it is more of a check list.

Chap 11: observational causal study
The methods in this umbrella are described as alternative methods to controlled experiment, to be applied in cases where controlled experiments are not possible. However, there are a plenty of examples where the results of observational causal study are far wrong from controlled experiment.
Honestly i cannot understand yet each methods in this chapter

10. Other complementary methods
This chapter lists other methods than experiment and their pros and cons. It creates some mental mappings in my heads when to use which methods, and many times we need to combine different methods to achieve the goal.
Controlled experiment maybe a demanding task to small companies without supporting platform.

Chap 8: institutional memory and meta data analysis
- organisation can keep track of the historical experiments they run and does meta analysis to get insights
- plenty of use cases

Chap 7: metrics for experiments and overall evaluation criterion
- experimentation metrics should be measurable, attributable (to features), sensitivity and timely. So not all business metrics can be used as experimentation metrics
- to combine metrics to OEC, we can normalize all metrics then weigh them
- some laws to be cautious with correlation and causal relationship.

Continuing chapter 6.
- how to formulate metrics
- evaluating metrics
- metrics can evolve
- some side notes about guardrail metrics and metrics gamification
So the above contents are more of listing. However i find it fun to read how ingenious humans can be in maximizing metrics by gaming around :))

6. Organisational metrics
Here i like the metric taxonomy chapter, where the metrics are divided to goal metrics, driver metrics, and guardrail metrics. The guardrail metrics are interesting category. We often think about what we want, but less often think of what we do not want and measure them to seek for mitigation.

5. Speed matters a lot
So this chapter focuses on performance controlled experiments, with the claim that performance is impacting a lot of other key metrics. Of course, we can safely assume that the higher the latency, the lower user satisfaction.
And maybe the author wants to make this an example how to perform such an A/B testing?

4. Experimentation platform and culture
What is needed to achieve the maturity of experiments:
- leadership
- processes
- infrastructure and tools: set up, deployment, instrument, scaling, and analytics.

3. Twyman’s law: interesting statistic is almost certainly a mistake.
- misinterpretation of the statistical result
- confidence interval
- internal validity: correctness if experiment results without generalising to other population or periods
- external validity: how much the results of a controlled experiment can be generalized to diff population and overtime?

2. A fictional example
- identify the experiment use case and goal
- establish hypothesis testing and statistical significance
- design the experiment
- run and collect data
- interpret the result
- make decision
I like the way the book starts.

Quantitative measure of the experiment’s objective (Overall Evaluation Criterion - OEC). Everyone knows that the experiments’ results need evaluating based on some metrics, but how to design the metrics takes a lot more efforts than we (I) assume.
- experiments can help hill climb to a local optimum based on current strategy
- or can suggest a better strategy to achieve higher win

1. Reading the first pages makes me realise that I underestimated the frequency of A/B testing in big tech.
- The book seems to focus on customer-facing product, especially software.