Status Updates From Apes in Lab Coats: Field Ob...

The Law of Unintended Consequences means that code-sharing principles intended to improve transparency, result in flawed methods sweeping through the scientific community faster than the Black Death in medieval London.

I had a colleague, Paula, studying jaguar (Panthera onca) who never got to see a jaguar in her entire three years in the field. Instead, the bulk of her research relied upon collecting and mapping jaguar droppings.

Fortunately for Paula, you can learn a lot from jaguar poo. Territories, diet, movements. She analysed the crap out of it.

Current reward systems combined with powerful cognitive biases - especially Confirmation Bias, Hindsight Bias, Observer Bias in conjunction with a strong Bias Blind Spot - favour progression-seeking behaviours over truth-seeking behaviours in scientific research and development.

More than half of scientists interviewed as part of the Nobel laureate process cite the importance of 'scientific intuition'. While their views may be interesting, there is massive Selection Bias and Survivor Bias at work. We only hear from the winners. It's the equivalent of winners of the X-Factor exhorting other contestants to never give up on the dream or 'go with your gut'.

Scientists are often quite intelligent.

This is a double-edged sword.

On the one hand, this allows them to rapidly adjust their world view based on their interpretation of new data.

On the other, it also means that they can quickly change their interpretation of new data based on their world view.

5. Exploring multidimensional spaces using low-dimensional tools may give rise to Hogarth spaces. These have the beguiling property of generating data to support scientists’ initial intuitions no matter how flawed.

6. Scientists encountering a string of Hogarth spaces may soon develop a misplaced confidence, bordering on arrogance, in their scientific intuitions.

When researchers understand and control for sources of variability, it becomes easier for others to replicate the experiment and obtain similar results. Reproducibility and replicability are essential for confirming the validity of scientific findings.

But is this how it works in real life? Within the four walls of the laboratory, what do scientists really think about variability? More importantly, what do they do?