Yuchen > Recent Status Updates

To reflect on last few chapters I found useful:

1. ch23, as always the author defines a standard pattern for reading the survey and find action items. This exercise is in general missing for my org of 100 peoples, which should have bi-annually survey.
2. building engineering hub: the author has downplayed the politics - have the team own the missions/critical area not required approval are not sufficient.

For these hiring and performance review chapters, I would just read "Scaling people", which provides a more systematic way to tackle the problems. The writing of these chapters are vague and shallow, hard to read, missing information and examples.

ch18 is very informative regarding to not burnout in case your standard is higher than the organization's.

1. acknowledge that holding other accountable might become "you two do not work together"
2. org has finite resources so low-standard work is a must.
3. be a role-model if you want other to change
4. recognize that other might be in different life situation as you.

ch17 reminds me ch12 "Task-Relevant Maturity" from high throughput management. In general you always inspect your reports' work. Depends on their proficiency level, you set different inspection interval.

Another thing to reflect is that inspection meeting is for leadership to build trust on engineers, instead of because leadership lack of trust of the team..

ch14 feels like a repetitive of ch5 (how to build values), ch13 (how to work with CEO, peers and engineering) and ch10 (how to meeting).

ch13 briefly discussed how to navigate the corp politics. I think overall the author had the same view as "The Staff Engineer's Path": understanding different points view and understand why people don't agree, why people points finger at you.

Ultimately it is easy to claim this is the best approach than exercise everyday because human's emotion exists.

I've constantly wanted to skip a lot of chapters. I think the book will be much better if different chapters can be grouped into different categories: how to get the job, how to establish engineering cultures/values, how to work with other orgs..etc

This is the third book I read from this author and all of them are having the same problems.

Reflect on Ch9: a lot of time whether to make a decision or not, I haven't thought about the "company, team, self" framework. It helped in retrospect as a lot of decisions were hard to make and debate about.

Although the author also provides a way for you to be energized, is to "prioritize some energizing work".

Ch8 provides a judgement for micromanagement:

1. Are you carefully considering the feedback from who closet to the work?
2. are you adding friction for the team who would have arrived at the same decision without your involvement?

ch5 is ok, it lists a few aspects make values useful: reversible for example, is interesting, makes the value more applicable.

Lack of example for ch6 makes it really dry and hard to read.

A poorly delivered chapter about planning and scratched surface of politics during the planning.

1. run Zero-Based Budgeting every year
2. how to spend time to validate the unscoped project: have a 10% allocation for validating projects.
3. Planning shall not award the least-efficient org. Whereas the author here does not provide a solution: "only CEO can address them".

Reflect regarding the ch3:

for the current corp job, we did not have a clear definition of fundamental rules: what standard development stack to use, how to approve exceptions.

Then we end up suffering about who should make decisions too. If what's team is doing follows the standard development stack, the team itself can make the decision. Right now the trade-off is not clear thus make people wondering..

Ch10 brings everything together

Step1:use RAG, see ch 6
Step 2:guardrails, see ch 5 for attach model, ch4 for qualify failures
Step 3:router&gateway, layer stacking, ch7
Step 4:cache, KV cache, prompt cache, SQL cache, semantic cache
step5: agent, ch6, write actions

Re user feedback
1. conversational interface makes it easier for user feedback
2. AI engineering is closer to product.

Ch9 is one of best. Inference optimization part has lot of insights.
1.quantization is more common/useful compared to pruning.
2.overcome decoding bottleneck: overcome decoding bottleneck: speculative decoding, inference with reference, parallel decoding,
3. attention mechanism optimization: KV cache size, kernel & complier.

Part of Ch8 feels like a repeat of the another book "Designing machine learning systems"'s ch4, especially about

1. how to handle lack of labels: use weak/semi supervision or active learning
2. data argumentation: perturbation, synthetic data

Some of the ideas are new

1. distillation
2. instruction data synthesis

Data evaluation and verification are challenging. It is common to use model eval method here too.

Part of Ch8 feels like a repeat of the another book "Designing machine learning systems"'s ch4, especially about

1. how to handle lack of labels: use weak/semi supervision or active learning
2. data argumentation: perturbation, synthetic data

Some of the ideas are new

1. distillation
2. instruction data synthesis

Ch7 is dense about fine-tuning. One good summary is "as model size increases, fine tuning become in practical as updating entire model's weight is inpractical".

One of the most important PEFT method is LoRA. It is memory efficient and modular: Easy to fine tuning multiple LoRA models based on the same base model.

Good chapter about RAG.

1. Term-based retrievers: BM25
2. Embedding-based retrievers: a lot of vector search algorithms
3. Ask the agent to use the function as tool: usually the tool function has a tool description, parameters section as agent's context. You can either let agent plan the work using real function name, or use a translator to convert language to function name.

Brief introduction for prompt engineering.

1. understand system and user prompt
2. best practice: adopt a persona, provide example, specify output format, break into subtask and use code to connect them, instruct model to use chain-of-thought

Ch4 talks about how to evaluate a model. I can tell that this chapter is more about trying different things and there is no formal procedure for it.

1. how to test hallucination: ask whether X has connection to Y for a model, when X has nothing to do with Y
2. a public benchmark might leak into model's training set
3. Use n-gram overlapping or perplexity to test data contamination

Solid Ch3.

1. why foundational models hard to evaluate? the language model component makes the evaluation open-ended. Unlike the ML's classification problems.
2. AI as a judge is useful, but human's evaluation is useful as it captures human's preference. Model can achieve perfect score in benchmark, human evaluation will never get saturated.

Informative chapter:

1. ch2 will be better if it explains the evolution from encoder/decoder RNN, to Dot-Product Attention, to scale dot product (LLM uses this).
2. the transformer blocks explanation is confusing.
3. discussion about pre-training, post-training and sampling are very informative. Consider read them again after finishing the book. Ch2 refer to other chapters a lot.

Chapter 1:

Good summary and great comparison between the ML engineering and the AI (application) engineering.

1. Multiple techniques to get the foundation models to generate what you want: prompt engineer, retrieval-augmented generation (RAG) and fine-tuning.
2. with foundation model availability today, it is possible to start with product, then invest in data&models when product show promises.

Good summary and great comparison between the ML engineering and the AI (application) engineering.

1. Multiple techniques to get the foundation models to generate what you want: prompt engineer, retrieval-augmented generation (RAG) and fine-tuning.
2. with foundation model availability today, it is possible to start with product, then invest in data&models when product show promises.