A cultura da experimentação: Como os experimentos nos negócios podem melhorar sua capacidade de inovação

by Stefan H. Thomke

When it comes to innovation, relying on intuition won't cut it anymore. Whether it's improving customer experiences, trying out new business models, or developing new products, even the most experienced managers often get it wrong. This is especially true in the online world, where predicting customer behavior is virtually impossible.

Managers can, however, discover whether a new product, service, or business model will fail or succeed--by subjecting it to rigorous experimentation. Think about it. A pharmaceutical company would never introduce a new drug without first conducting a round of experiments based on established scientific protocols. Yet that's essentially what many companies do when they roll out new products and services.

As Harvard Business School professor Stefan Thomke shows in this eye-opening and essential book, the "best guess" approach to innovation is changing fast. There are now leading companies that conduct more than ten thousand online controlled experiments annually, engaging millions of users. These organizations have discovered that an "experiment with everything" approach has a big payoff, giving them a considerable competitive advantage.

How can you do this at your company? Leaders and managers need to create an "experimentation organization" that masters the science of testing and puts the discipline of experimentation at the center of the innovation process. It used to take companies years to build the infrastructure and develop the expertise to run hundreds of experiments each day. But Thomke shows how, with advances in technology, these capabilities are at the fingertips of almost any business professional. By combining the power of software and the rigor of controlled experiments, today's managers can make better decisions, create better customer experiences, and generate huge financial returns.

Filled with engaging and instructive stories of leading experimentation organizations, Experimentation Works will be your guidebook to a truly new way of thinking and innovating.

Genres: Business, Nonfiction, Management, Entrepreneurship, Science, Design, Leadership

328 pages, Kindle Edition

First published February 18, 2020

Reviews

[Lukas Vermeer · Rating 5 out of 5]

There are lots of books that explain in detail how to design and execute a randomized controlled experiment. This is not one of those books. Instead, this book sheds some much needed light on the questions of why and how organizations can utilize experimentation at scale to drive evidence-based customer-centric innovation.

“Large-scale testing is not a technical thing; it’s a cultural thing that you need to fully embrace. You need to ask yourself two big questions: How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you? And if the answer is that you don’t like to be proven wrong and don’t want employees to decide the future of your products, it’s not going to work. You will never reap the full benefits of experimentation.” (David Vismans, Chief Product Officer of Booking.com)

[Guy Taylor · Rating 5 out of 5]

I have a very biased view of this book, and the review may not be helpful to all. As the current Director of Experimentation at Booking.com, the book was useful to me to get a lot of the historical context of Experimentation at Booking. More so I really enjoyed the principles and culture super structures that the books holds up, it made me think about the non-technical and people orientated pieces of Experimentation. This book is not a how-to, or a technical read, but it is wonderful to read as a ‘how to think about thinking’ type book.

[Sarah Parker · Rating 1 out of 5]

There are lots of business books that still manage to be engaging page turners, this is not one of those. It’s incredibly boring and repetitive, and it feels 10 times longer than it actually is.

As the title suggests, the focus of this book is proving that experimentation works by discussing lots of examples from enterprise giants like Microsoft, Bing, Pinterest, Booking.com, and State Farm. The main problem I had with this book is that it seems the target audience is large enterprise companies with the size and ability to scale enormous experimentation departments. Thomke discusses how convincing the entire company particularly management of the importance of experimentation is the most difficult step, but in 2023 I have an incredibly hard time believing that this barrier exists at most thriving and successful enterprise businesses. (Most examples he provides are from around 2018, and things have changed a lot since then)

While I did appreciate the discussion about the challenge of widespread cultural acceptance versus technological capabilities, I still think that Thomke has wildly overestimated the number of people and businesses who do not already value experimentation and aspire to use it as a tool for business growth.

Overall, I wish that there had been more of a focus on how smaller businesses, start-ups, and teams with small budgets can still harness the power of experimentation to drive business growth. Instead, by exclsively using examples of companies that run hundreds if not thousands of experiments per year, he implies that experimentation is only valuable at a large scale, which simply is not true. I think that this book would have been more valuable back in 2018, but now the conversation should pivot to how new low-cost A/B testing tools can help small and mid-size businesses grow and compete with companies that have already integrated A/B testing into their business strategies.

The one thing I did really enjoy about the book, is that the quotes dispersed throughout were very insightful and thought provoking.

[Greg Hawod · Rating 5 out of 5]

Leading innovation in organization requires experimentations. We wouldn’t know if our ideas will stand the test of even a short time if we haven’t tested it using the scientific method.

This book is full of exciting and deep insights on business experimentation. It lays the foundation of what it means to conduct business experiments as well as the possible hindrances towards its adoption.

The books also helps organizations dealing with so many uncertainties to develop the discipline of experimentation so they can isolate data noises from the critical insights needed to make tractions of their goals.

[Tugce Gencoglu Ljungholm · Rating 4 out of 5]

The book is about how companies adapt different experimentation techniques and how managers/senior leadership can create an experiment culture by taking 20% success rate in the experiments into account. If you are a data analyst/scientist already running a/b tests, this book can give a strategic perspective of the experiment culture.

[Andreas Aristidou · Rating 4 out of 5]

Overall I found this to be a great guideline – a must-read for any business/organization thinking about implementing experimentation. The author takes us through countless examples of companies and organizations who have embraced an experimentation culture. Throughout those examples, he exposes readers to the hurdles and the benefits of becoming an experimentation organization. His vast experience in this area shines throughout the book.

While the book touches upon the math and statistics of experiments, it’s not the main purpose. The book is structured around several companies and high-level executives who have transformed their organizations to embrace experimentation. As such it looks at the big picture of experimentation rather than its technical details. In what follows I will attempt to summarize each chapter by discussing the key points, quotes and examples from each chapter.

Read my in-depth summary of the book here:

https://aristidouandreas.com/book-rev...

[Alexandre Blanco · Rating 5 out of 5]

After reading this book you realise why companies such as Zara, Amazon or Booking became leaders in their markets and especially how top performers are adapting and shaping the market by trying and trying in a high-speed way under today very uncertain and changing scenarios. They do it through extensive experimentation. Prof. Thomke describes in precise and understandable words for non-scientist readers the experimental design method (deduction, induction, abduction, the use of controls…) and how to set it up in house, not focusing in R&D only, but in other company units which would integrate Marketing, Sales, Customer Service… which can easily be taken and run by all type of companies, not exclusively by internet technology firms. What I find more valuable for those managers who want to implement an experimenting culture or those Business readers is the holistic approach Prof. Thomke provides describing business strategy, the identification and “conversion” of key stakeholders towards Experimentation and the procedure and tools a to make it happen and even introduce a real corporate culture change taken in the book. Furthermore, the author intertwines practical cases such as the New Zealand America's Cup sailing team with research findings with concept/idea development and traps the reader to go to the next chapter immediately.
Experimentation works is one of the reference business books of the last years and a must read not only for Top Management Executives seeking Disruption, sustainable competitive advantages or an agile system to generate more innovative solutions, but for those managers who can benefit in their marketing, sales, IT or operations arenas as well as for those business readers who want to understand the advantage of those companies performing a high number of experiments in a short period of time to adapt and even shape today’s ultrafast changing global markets.

[Danilo Weiner · Rating 3 out of 5]

Gosto de livros técnicos mais práticos, que ajudem a resolver problemas mais táticos do dia a dia. Obviamente um livro chamado “A Cultura da experimentação” já dá pistas que vai falar de modelo mental, mudança cultural e outras coisas mais de longo prazo. Dito isso, o livro me surpreendeu posivitamente, ao abordar exemplos de empresas que foram bem sucedidas na implementação da cultura da experimentação, descrevendo com algum detalhe algumas práticas de sucesso.


Além disso, como todo bom viés de confirmação, gostei de ver o autor discorrer sobre o conceito exploitation / exploration, a provocação sobre o Big Data como a bala de prata e a necessidade de se aprofundar o estudo de correlações para realmente estabelecer uma relação de causa e efeito entre as coisas.


Acho que um ou outro exemplo citado como melhor prática (Ex: Bing) não transparece ao mercado como algo a ser seguido, mas saio da leitura com algumas boas inspirações...

[Paulo Peres · Rating 5 out of 5]

O livro é um bom exercício de forma técnica sobre como pensar, fazer e implementar a Experimentação dentro da empresa. É fortemente baseado no estudo do autor da empresa Booking.com, mostrando também muitos exemplos da Microsoft.

Um dos maiores ganhos que o autor nos entrega é a mudança de mentalidade que Testar / Experimentar nos traz: o incrementalismo é importante para a inovação! Testar sempre e de forma rigorosa e bem montada, traz sempre lucros. Small and continuous change! Tanto para uma empresa ambidestra ou não, experimentar (a experiência do cliente, backend e modelos de negócios) é importante para ganhar competitividade através de números (data-driven).

O livro poderia ter um toque de design com mais diagramas, mais esquemas visuais. Isto ajudaria o leitor a fixar a leitura mais tempo.

[Alex · Rating 5 out of 5]

Once in a while, I find myself becoming jaded and cynical towards experimentation. Not because of experimentation itself, but rather the seemingly Sisyphean task of promoting and building a culture of experimentation in a company that seems not to want it. This book reminded me that experimentation is fun, awesome, necessary, and it works (and one must imagine Sisyphus happy). It's not overtly tactical, but it covers the broadest possible landscape of organizational cases I've seen, including non-digital-native companies like State Farm as well as the well known examples like booking.com and Pinterest. I will, as a negative, this book tends to gloss over the faults and gaps that clearly exist within these case companies (you can easily glean this by talking to anyone who works there). But as a case for managers and executives to adopt experimentation, top tier book.

[George Mount · Rating 4 out of 5]

Relatively good introduction to business experiments. In particular the author did a great job explaining the organizational/cultural implications of experimentation. However the examples were nearly entirely taken from digital business (It read like a pamphlet for Booking.com at times.).

The author himself stated one of the "myths of experimentation" was that it couldn't apply to "analog" business, but didn't go into nearly as much detail on them (Did I mention the author enjoys writing about Booking.com?).

All that said, this book offers a great take on the compatibility of analytics and innovation and (roll credits) the power of business experiments.

[samkrunch · Rating 3 out of 5]

If you're a data scientist looking to brush up on A/B testing, this book is just way too high level to be useful (even if you were okay with skipping over some of the technical stuff). However, if you're trying to convince your product manager/MBA coworker to rely more on data from controlled experiments over their intuition, you could recommend this book and it might help. The anecdotes sprinkled throughout the book of various tech companies winning by implementing experimentation get pretty repetitive and boring, but I can imagine that some people may need to see a million examples of the same thing working over and over again before they can be swayed to try it out themselves.

[Max Currier · Rating 3 out of 5]

Very focused on digital experiments, and not just in the chapter dedicated to that topic. Needed more discussion of "experimentation works" in non-digital arenas.

Also, the book emphasized the importance of technical rigor in executing good experiments, but did not address those technical topics; for example, it emphasized the importance of Bayes' Theorem but made only a vague, high-level description of it (and there was no technical appendix). Managers who rely on teams to deliver them statistical analyses will struggle to interpret it or ask good questions if they are not reasonably familiar with the technicals of what they're consuming.

[Jacob · Rating 4 out of 5]

Fairly introductory and not a lot of new information here but a good overview of experimentation/why it works. Reminded me of how much I love the topic. Our intuitions are often wrong and real change usually takes place through small, iterative product changes. Chapters 3 & 5 are worth coming back to, especially the case studies around booking.com and Microsoft. I was less interested in the sections about how to make your organization more experiment-driven because this is already the case at most places I’m interested in working for.

[Luiz Gustavo Amorim · Rating 5 out of 5]

This book shows how scientific methods are being applied to sustain value creation via AB testing, computer simulation, or even surveys.

Every chapter is well illustrated with up-to-date benchmarks and practices regarding people, processes, and tools from digital (booking, LinkedIn, Microsoft, Pinterest) and traditional (Petco, Kohl, America’s Cup Teams, IBM) companies.

You will also make very good use of the long list of notes and bibliography. I would suggest a combo with Ron Kohavi’s and Ming Zeng’s latest books.

[Cassie · Rating 5 out of 5]

Great for marketing professionals and beyond

I read this book because I listened to a podcast about experimentation a few months ago. Since I recently started working at a digital marketing company I thought it would be useful to dig deeper and read the book. To be honest, it was a bit boring for me so I kind of power-read it, but I certainly gained a few golden nuggets of information that will help me excel as an HR manager in a marketing company!

[Jaime Moncada · Rating 4 out of 5]

Cuando se trata de mejorar la experiencia de los clientes, probar nuevos modelos de negocio o desarrollar nuevos productos, incluso los gerentes más experimentados a menudo se equivocan.
Este libro brinda una perspectiva y un marco de trabajo sobre cómo implementar una cultura de la experimentación y del testeo sobre todas las iniciativas utilizando el método científico en la toma de decisiones empresariales.

[Sukhveer · Rating 4 out of 5]

While overtly prescription at times (e.g., 7 attributes, 5 lessons etc.), the fundamental premise of using to scientific method to making all decisions including the business decision makes for rather persuasive reading. There was a bit of an over reliance on Booking.com example as many will find it hard to translate the methodologies to more complex and regulated industries such as healthcare. Thought provoking nonetheless and well argued thesis.

[Robbie El · Rating 4 out of 5]

This was my second read of Experimentation Works. I’m a scientist-practitioner in healthcare, so experimentation is a part of my everyday toolkit. That said, it’s inspiring to think of experimentation in the context of business, especially if like me, most of your experience is more clinical in nature. This is a phenomenal book if you’re eager to learn more about the empirical process applied to the workplace.

[Isa Lima · Rating 4 out of 5]

O livro é muito instigante e provoca o leitor que atua em organizações a iniciar processos de experimentação no seu contexto. Traz reflexões sobre cultura, armadilhas pra se evitar e elementos processuais para ter atenção. É uma pena que essas grandes obras sobre negócios sejam sempre focadas apenas em gestores e não dêem aos analistas ao menos dicas de como iniciar ou provocar os gestores sobre a importância desses temas.

[Dianna · Rating 2 out of 5]

Advertised as a book about experimentation in business, it was focused on large companies with lots of traffic and data, such as Amazon and Booking. There wasn’t a clear way for those in other types of businesses to apply these methods. Some good information about applying innovation and gaining acceptance throughout an organization.

[Emmanuel · Rating 4 out of 5]

A great book to read if you are looking to start up an experimentation culture on your team/company. Lots of examples given mostly in the context of bigger companies but a lot of applicable principles to smaller companies too. Helps you understand the power of experimentation and better convey it to people who might not.

[Reid · Rating 3 out of 5]

Good motivation and case examples from big tech. Very high-level without direction on 3rd-party tooling, integration or building internal infrastructure to support testing. No mention of legacy or resource constraints. Blue sky.

[Kinza · Rating 3 out of 5]

Could have been better if the author included examples outside of software tech and engineering! Fell flat in the last third. By that time, I’m certain readers were done in their experimentation with this book.

[Ravi Warrier · Rating 2 out of 5]

Wish the author had suggested how to experiment when you are not a multi-million/multi-billion dollar company.

Everything is doable when you have tons of resources to spare. The better trick would be figuring out how to when you don't.

[Shabana bi · Rating 2 out of 5]

Great

[Lee Barry]

I did only a cursory reading, but I can see how this would be of great interest to people that are involved in business innovation.

[Can Karakulak · Rating 5 out of 5]

Best business book I've read in 2020 so far.

[Mark E. · Rating 4 out of 5]

Practical examples of how to improve experimental approaches, particularly for consumer products

[Jerome · Rating 2 out of 5]

Too academic. Useful for people working in tech