Cautionary Tales of Complexity and the Dangers of Jumping to ConclusionsIllustration by Anni Nöteberg

A/B testing is becoming increasingly popular in digital product development. Early feedback on the choice of direction may save you a lot of time and money. However, be careful about drawing too far-reaching conclusions from your test results.

Imagine that for one day we let 5% of our retail’s online customers test a new feature. We notice that those who have the new feature (the test group) spend...