I am convinced that developers and technical managers should strengthen their refactoring practice. Indeed, unless you change companies every couple years, each time starting with a new project from scratch, large scale refactoring will be part of the life of the teams one day or another. It can be risky (as in killing you company or more frequently costing unnecessary time and energy).
Martin Fowler's classic book Refactoring is more about code smells and examples of code rewrites, so I wanted to learn more about the higher level aspects with Refactoring at Scale.
The book details throughout the chapters a methodology for refactoring projects. It comes from the author's experience, especially at Slack, which has, with its phenomenal growth, had to address a number of them.
The skeleton of the methodology is detailed in the book (after a first introduction part), consisting of:
Planning phases
1/ Measuring a starting state: gathering metrics assessing the problem, why the refactor must be executed and when is the best time (eg code complexity metrics, formal/informal documentation, version control, pain points chatting with colleagues...)
2/ Drafting a plan: defining an end state for the metrics, choosing a strategy, defining and estimating milestones, choosing a rollout/rollback strategy. At that point, you should have a plan to share with other teams for reviewing. It should gather the why / when / who / what, list all phases and for each one: a todo list, an estimate, start metrics and goal metrics.
3/ Getting buy-in
4/ Building the right team
Execution phases
5/ Communicating: communicating both with and outside the team during the project
6/ Executing: this chapter describes a technique used at Slack called "dark mode / light mode" to compare pre-refactor and post-refactor behavior. It consists in several phases. First, during dark mode, both old and new implementations are called, the results are compared / logged if different, and the results from the old implementation are returned. It helps finding bugs/regressions in the new implementation. Second, after bugs are fixed, next phase (light mode) kicks in, starting with a subset of low risks users then progressively more and more of them. In this phase, both implementations are still called / logged but this time results from the new implementation are returned. Finally, old implementation is removed.
7/ Making the refactor stick: workshops, office hours, documentation, linters to make sure the refactor is widely known in the company and in some cases well used
What I liked
- the principle of using metrics to objectify the success of the project (classic in product management, less in purely technical projects)
- the dark mode/light mode pattern, simple and beautiful (see on this subject the "scientist" library developed by GitHub)
- the two cases studies at the end of the book (a database schema change and a database engine migration). They are much more detailed than blog posts, more transparent and authentic also about the mistakes made too (remember that many technical blog posts have a marketing objective of recruitment... and forget to mention some details)
What I didn’t like
- Some parts were too long, some too basic
- I was hoping for some material on the thorny subject of "big rewrites", technical debt management, different migration strategies
- The passage on how to "sell" the project to the manager inviting giving an ultimatum and threatening to leave the company. For me, this is a huge NO, it can totally break confidence.
Highlights
"Confidence in your ability to refactor allows you to lean toward action and start building a system sooner, well before you’ve developed a strong understanding of all the moving pieces, pitfalls, and edge cases."
"The most common culprits behind tech debt are limited time, limited numbers of engineers, and limited money. (...) Although it can be easy to point a finger at the original authors of the code and admonish them for making decisions that appear suboptimal today, it’s important to remember that they were operating under serious constraints."
"Be cautious of conversations in which colleagues say something like, “While we’re at it we could…” or “I’ve always wanted X to also handle…”. Unless you’re well-versed in the art of saying “no,” you might end up agreeing to do more than you originally anticipated."
"Most large-scale refactors have sizable milestones consisting of tedious, repetitive work. (All of the examples in this book to date have one or two lengthy, monotonous steps.) These milestones tend not to be exceptionally challenging or engaging; they’re dull but necessary. When the team needs to execute these stages, usually the project starts to feel as though it has slowed to a crawl. Teammates can be more prone to burnout during these stages, but having a group of individuals you can lean on, with which you can share your frustrations, can make a world of difference."
"I highly recommend developers begin their code archeology expedition before they begin to execute their refactoring effort, because the added context can give a different shape and direction to the project."