Escape from Shadow Physics: The Quest to End the Dark Ages of Quantum Theory

by Adam Forrest Kay

An expert physicist argues for a revolutionary new understanding of quantum mechanics  The received wisdom in quantum physics is that, at the deepest levels of reality, there are no actual causes for atomic events. This idea led to the outlandish belief that quantum objects—indeed, reality itself—aren’t real unless shaped by human measurement. Einstein mocked this idea, asking whether his bed spread out across his room unless he looked at it. And yet it remains one of the most influential ideas in science and our culture.    In Escape from Shadow Physics, Adam Kay takes up Einstein’s reality isn’t mysterious or dependent on human measurement, but predictable and independent of us. At the heart of his argument is groundbreaking research with little drops of oil. These droplets behave as particles do in the long-overlooked quantum theory of pilot waves; crucially, they showcase quantum behavior while being described by classical physics. And that classical-quantum interface points to a true understanding of quantum mechanics and a reasonable universe.    A bold and essential reset of the quantum physics field, Escape from Shadow Physics the kind of true scientific revolution that comes along just once—or less—a century. 

Genres: Science, Physics, Nonfiction, History, History Of Science

423 pages, Kindle Edition

Published June 18, 2024

Reviews

[Simms · Rating 5 out of 5]

4.5 stars, rounding up. Adam Forrest Kay is really speaking to me here, as someone who has always found quantum physics to be deeply unsatisfying (and confusing). His point is, essentially, that in every case in science where a statistical model was developed to describe something, the statistics were reflecting a deeper physical mechanism that we only later were able to discover and understand -- yet, when that dynamic plays out with quantum physics, the quantum physicists insist that there is no deeper system, no "hidden variables," and that reality simply is probabilistic. But why, Kay argues, should that possibly be true only in the case of quantum physics?

Kay does a good job here explaining how the "Copenhagen interpretation" came to be the dominant scientific explanation of quantum physics, with a lot of historical parallels of other moments in scientific history where the dominant explanation of a thing was wrong (e.g. phlogiston, the caloric theory of heat, miasma theory). His message to quantum theorists is to be humble: plenty of scientific certainties have been overturned before. Kay does seem to have a bit of a chip on his shoulder about this, as one of the minority of scientists who are working to try to determine those "hidden variables" that the Copenhagen interpretation dismisses. Along the way he spends a good bit of time describing some quite exciting developments in what are called "hydrodyamic quantum analogues" where wave/particle behavior can be exhibited through purely classical, fluid-dynamical means, and while I confess I don't totally understand what he's going on about there I do love to hear that people are doing research that might "classicize" quantum physics.

Thanks to NetGalley and Basic Books for the ARC.

[Márcio · Rating 5 out of 5]

(...) It is so much safer and easier to take the current majority view and report that to the public. But this book is about dissent, about outsiders, and about growth in human knowledge, sometimes even at the expense of the majority. So, damn the torpedoes, full speed ahead!

I must admit that Adam Forrest Kay is more than a voice of dissent and what he does in this book is proof enough that his main goal is to shed light on the state of quantum mechanics for the last (almost) 100 years since Bohr and his followers decided that the Copenhagen Interpretation was all there is to it. With that, Physics has stuck and not much has happened in the last hundred years to preserve the ideas of its positivistic idealizers.

Still, and gladly, some believe quantum physics is a kind of approximation and there is more at a deeper level to be sought and found/thought and brought up than what quantum theory can describe up to this day. Kay does it greatly here in his Escape from shadow physics, and I can also indicate another such book like Adam Becker’s What is Real?: The Unfinished Quest for the Meaning of Quantum Physics . Both wrote books that allow any layperson to understand how it all evolved and what the state of things is right now.

It is just that maybe Kay is more audacious in the way he faces those in the academia and scientific world who don’t seem to accept dissent, as we’ve seen through the ages. These, the ones who usually hold Science hostage at the expense of the growth of knowledge are also the ones greedy for public money to advance their ideas, most at the expense of other researchers and lines of thought as Sabine Hossenfelder keeps bringing up.

[BJ Richardson · Rating 4 out of 5]

Adam Forrest Kay's Escape from Shadow Physics is a critique of the prevailing interpretations of quantum mechanics, particularly the Copenhagen interpretation. The author challenges the idea that quantum phenomena are inherently probabilistic and observer-dependent, advocating instead for a deterministic and objective reality. He draws inspiration from pilot-wave theory, first proposed by Louis de Broglie and later adopted and expanded by David Bohm. This theory suggests that particles are guided by underlying waves, and offers a more intuitive understanding of quantum behavior.

Central to Kay's argument is the experimental work of Yves Couder and Emmanuel Fort, who demonstrated that oil droplets bouncing on a vibrating fluid surface could mimic certain quantum behaviors. These hydrodynamic analogs provide a tangible model for phenomena traditionally considered uniquely quantum, such as interference patterns. Kay posits that such classical systems can offer insights into quantum mechanics without resorting to abstract probabilities or observer-induced realities.

Kay's writing is both rigorous and accessible, weaving historical context with philosophical inquiry. He challenges readers to reconsider long-held assumptions, stating, "We are told to believe there is one—only one!—area of probability theory in which the concepts grown by Pascal and Fermat apply, but the normal way of thinking about those concepts is wrong. This area is quantum physics." By highlighting the parallels between classical and quantum systems, Kay encourages a reevaluation of the foundations of physics.

While Escape from Shadow Physics presents a thought-provoking perspective, it is not without its critics. Some argue that the hydrodynamic analogs, while illustrative, cannot fully replicate quantum phenomena, particularly in experiments like the double-slit, where replication has proven challenging. Nonetheless, Kay's work serves as a valuable contribution to the ongoing discourse on the interpretation of quantum mechanics, urging both scientists and lay readers to question and explore the nature of reality.

[Katie · Rating 5 out of 5]

A love letter to the physics community ... but likely to cause a stir in some circles.

Kay does his best to bring the gentle reader into the realm of physics and especially quantum mechanics. I admit, even as a PhD holder in the sciences, I was lost at several points. There's just so much one can do, I suppose. But I urge the reader to carry on. Concepts and what they're all about are repeated and re-introduced. There are many colourful metaphors and examples, and several interludes that may seem utterly disconnected from the main thread, but end up masterful and compelling analogues.

For me, the real treat was less about the physics and more about the love letter. I'll write it again: this is a love letter! The penmanship, which had me in equal parts riveted and in stitches! The tale, which navigates a long tapestry of personal and professional histories ... perhaps the most engaging parts were about the relationships among these people ... from rivalry to camaraderie and everything in between ... somehow Kay has unearthed the diaries (or near enough) of some of the greatest actors on the physics stage and bared it all for us. All meticulously referenced.

I am conscious of being only an individual struggling weakly against the stream of time. But it still remains in my power to contribute in such a way that, when the theory of gases is again revived, not too much will have to be rediscovered.

Here is Boltzmann, writing in 1898, (Vorlesungen über Gastheorie), at the mercy of humanity's greatest weaknesses when it comes to scientific practice. Indeed, our faults have afflicted every major player one can think of ... a sobering thought, but also a comforting one. Yes, I had no idea that Bohr was such a godlike figure, but also that Einstein was such a joker. Science is a human enterprise, and physics is no exception. The community continues to be divided on "the facts," which some believe are decided and others, not so much. Kay has his own opinion. "Positive is rat poison" he writes at one point. The main point, rather than anything about physics and quantum theory, is that people like boxes. And when people make a box, they want to keep it ... at all costs. And this is anti-scientific. Yet, as Kay writes, "real science is done by real animals, it is an animal institution, and as such is subject to many unscientific forces." Yup, we're power-hungry oligarchs with no shortage of bias! Kay deftly highlights some of the most notable of such blips in quantum physics. I had no idea the history of fundamentals was so exciting ... but why not, when we're all beasts blundering our way through the natural world together?

On that note, the history of physics is very white and very male, but Kay does his best to incorporate women, speakers of other tongues, and non-Westerners. Still, there's a lot of language that reinforces the historical and contemporary wiener fest that is physics. For instance, Kay updates one hypothetical to be in a lab rather than a cafe, but it carries the same tropes and might even be worse off, since now we're bringing cisheterosexuality into the workplace. And, as per usual, things like atoms are anthropomorphized as "little guys."

I was also a bit miffed that Kay in one sentence denounces non-scientists who "like to appropriate scientific thought" after giving an example and going on to provide even more of scientists, especially physicists, indulging in pseudo-scientific malarkey. Call all the spades a spade!

Nevertheless, this is a gripping narrative and nice introduction to quantum physics: the people, the problems, and the pandemonium. We still have much to learn, about the natural world and about ourselves.

Thank you to NetGalley and Basic Books for the advance copy.

[Africandreamer · Rating 5 out of 5]

Absolutely brilliant.

Now I know why PhD-level English Literature and Mathematics are perfect bedfellows...the ability to tell a story in wonderfully clear and highly engaging prose about a subject that will otherwise leave 99.9% of the population floundering.

One could draw the conclusion from this book that AFK is simply a grumpy and very sore member of the losing side of the hidden variables debate that has raged for 100 years (or not in the minds of the winners who have so convinced themselves that they have won that there is no more debate). That would be entirely the wrong conclusion. ADK clearly doesn't know the answer to this debate, rather he passionately and entirely convincingly makes the case that the conclusion that quantum physics is, at its most detailed level, a statistical process for which there is no underlying explanation, just cannot be right. His position throughout the book - supported by a grand tour of scientific endeavour through the ages - might be summed up as "My fellow physicists...where art thy curiosity?". I'm neither a physicist nor a mathematician, but I found ADK's argument for this utterly compelling.

I was for an instant put off by the title, thinking this would be a heavyweight text that would take weeks to ingest. It's not and it didn't. Started on a Saturday morning and finished the next day. Gripping the whole way through. Just read it!

Looking at the other books I've read on quantum physics - strictly as a layperson who enjoys feeling just a tiny bit more educated (!) - this one is easily the best one that I've read. Maybe that's because I just agree with the entire premise of the book that scientists (well, all humans, really), should be curious. If you are thinking of another book on the topic, I'd recommend Philip Ball's Beyond Weird...very well written (if not a somewhat more cerebral read) and includes an excellent demolition of the Many Worlds Interpretation.

Incidentally, the Copenhagen Interpretation got me thinking. Now that I know that the Moon doesn't exist when no-one's looking at it, how about the following:

- I can see the Moon whenever it is out, while my father, who is very nearly blind, can probably only see a full moon and, even then, probably only the fact that a patch of the sky is brighter than its surroundings.
- We're the last two people on Earth
- I'm on one side of the Earth - away from the Moon - and my father is on the other side of the Earth from which the Moon would be visible to me if I was there but isn't visible to my father because he has terrible vision...according to the Copenhagen Interpretation, apparently the Moon doesn't exist because it hasn't been seen.
- We swap places and the Moon now does exist, because it has been seen.
- Hey presto, the existence of the Moon depends on the visual acuity of the observer.

Next experiment:
- The other night, I could only see a tiny slice of the moon...does this mean that the part of the moon that I can't see doesn't exist because I can't observe it?

Next experiment:
- I carefully fill the moon with a lot of explosives, move to the other side of the world and blow it up from a position where I can't see it (assume the explosion pushes the Moon fragments substantially sideways - these are 'shaped charges') - and possibly even into an orbital ring.
- Clearly, because I wasn't observing the Moon when I blew it up, it wasn't there, so there will be no impact on the tides experienced on Earth.

Nonsense, of course.

Where will all of this end?

[Benjamin Curry · Rating 4 out of 5]

A fascinating book. Kay worked alongside Prof. Bush on Hydrodynamic Quantum Analogues, that seems to point compellingly to a deeper, causal and eminently comprehensible reality beneath present day quantum mechanics.

The book is more than that, however. It is a conscious materialist defence of the scientific method against the penetration of idealism in the form of the Copenhagen Interpretation. The author takes a historical approach, looking at everything from the theory of heat, to the miasma theory of disease propagation, to plate tectonics. He shows, in an engaging way, how science progresses, how theories we now take as correct were held up by scientific establishments and specifically by the philosophical views prevailing at any one time.

The strongest aspect of the book is that it directly philosophically takes on positivist idealism, which underpins the present Copenhagen Interpretation. What I wasnt aware of was the influence of positivism on other scientists, like the influence of Comte on Fourier and the uncannily similar (in fact identical) arguments that were used to deny the onward advance of science in earlier epochs.

There are two weaknesses to the book. Despite taking a historical approach, Kay doesn't get to the bottom of why certain philosophical ideas seem to hang over some epochs more than others, why some periods in history are periods of general advance of science, others of general decay. He often says that the Copenhagen lot accept determinism (ie lawfulness, causal relations) in all areas of the universe except the quantum realm. It should be said that society is also lawful, but Kay doesn't get to the bottom of those laws.

The second weakness, I feel, is that Kay often uses the terms "classical" and "mechanical" rather loosely and non-philosophically to mean the restoration of physics with "mechanism" (ie cause and effect) as oppose to the indeterminism. And the one mention of "dialectics" is rather disparagingly used in connection with Bohr's complementarity (the bare assertion of contradictory properties of matter, like particle and wave behaviour, while simultaneously denying that they form a unity in the deeper processes at play, which is not dialectics but mere inconsistent thinking).

These do not detract from my appreciation of the book, from which I learnt a lot and which I think is excellent and deserving of a wide audience. There are a few points where it gets a bit technical, but they are brief and not too overwhelming.

[Marie Belcredi · Rating 5 out of 5]

Adam Forrest Key has PhD's in Literature and Physics so has all the ingredients needed to write a magnificent book about quantum physics. I left off trying to understand physics when I was told that there was no explanation and our macro minds could never understand. Einstein always held on to the belief that there was an explanation and he and Bohr never settled this between them. As the well known saying of Einstein "God does not play dice" is famous for. Bohr was a bully and he succeeded in making sure that his Copenhagen Interpretation remained the dominant one for much of the the 20th century.
Key gives many examples of scientists through the centuries grappling with observations that they could not explain. One interesting parallel was our view of comets. In Istanbul in 1577, the sultan's astronomer suggested that the comet foretold good tidings. This turned out to be very false when several dignitaries died and plague affected them. That was the end of the Observatory in Istanbul.
For the same sighting, a Lutheran bishop suggested that a comet was "the thick smoke of human sins.. kindled by the hot and fiery anger of the Supreme Heavenly Judge. A wit remarked that if this were the case, comets would never be absent from the sky.
This book is full of analogies to make it easy to understand the arguments but the history of science reveals much about human nature. Why do we often not recognise true progress in science ? How was Bohr able to stop any real progress in science and prevail over de Broglie after he had given his paper at the Solvay conference?
This book clearly and with a sense of humour describes for the reader the current thinking in Physics and thankfully there may be a rational explanation after all of quantum observations like entanglement. We just dont exactly know it yet. Quantum mechanics in not complete.

[H. Alesso · Rating 5 out of 5]

Illuminating the Quantum Shadows

Adam Kay's "Escape from Shadow Physics" challenges conventional quantum mechanics. It offers a fresh perspective for our understanding of the universe's fundamentals. His approach is both intellectually stimulating and thought-provoking.

Kay traces the wave-particle debate through history to the present. His critique of the Copenhagen Interpretation is particularly insightful, convincingly arguing that it contains dogmatic elements that lack empirical justification. The book shines when it explores alternative interpretations, especially the pilot wave theory. Kay's discussion of classically bouncing oil droplets demonstrating wave-particle duality opens up exciting possibilities for understanding quantum phenomena through classical means.

The engaging narrative balances historical, philosophical, mathematical, and physical ideas. While favoring a deterministic interpretation, Kay presents various viewpoints fairly.

While not providing definitive answers, "Escape from Shadow Physics" challenges us to think critically about established theories and opens up new avenues for exploration in theoretical physics.

[Christie Wessels · Rating 4 out of 5]

This book is above my level and I had to push myself through the slog of mathematical explanations and physics concepts that are beyond me, but the basic premise is full of hope. The author challenges the accepted view of quantum mechanics, illuminating the flaws in premises and logic, the physicists throughout the 20th century who disagreed with the consensus and were ignored or ridiculed, and offering a possible new way to see and explain the supposed randomness of the movement of electrons at the quantum level.
I love the way the author traces the history of quantum mechanics and many other fields of study, and ends the book with this: "Often, the correct idea could have been had at any time. It did not depend on any particular knowledge. Rather, it depended on taking the right viewpoint. It is viewpoints that blind us or let us see. It can take a hundred years for a person with the right viewpoint to come along and think heartbreakingly obvious thoughts. "

[frank font · Rating 5 out of 5]

Keys and Recommendation

It’s annoying how so many reviews of this book seem to miss the point of the book. Have the reviewers actually read every page? I suspect too many of them have skipped pages assuming what was not there.

I’ve read it all from cover to cover and let me tell you it’s a terrific book.

Here are three key punches that will knock you back:

1. Extensive science history showing parallels of the mistakes some of the sharpest minds in philosophy and physics are making today.

2. Compelling arguments undermining the claims that there are any valid proofs that quantum mechanics is complete. ( the Copenhagen interpretation has no legs of proof )

3. The book does not claim that reality is deterministic, nor does it claim that it is non-causal —- the major point of the book is that there is currently no valid reason to stop looking for underlying mechanisms that model and give insight into what our statistical models of QM currently surface.

There is nothing irrational or weak that I read in the pages of this book. Bravo to the author. Terrific presentation.

[Douglas Summers-Stay · Rating 4 out of 5]

A thoughtful argument for the fluid dynamics analogy for quantum phenomena and the pilot wave idea. The key chapter is the one on Bell's inequalities and why they do not disprove hidden variable theories, but that is the one chapter I found unsatisfying. I don't understand what he is proposing as the alternative interpretation of Bell's results. If I take his word for it that there is a way around it, then the rest of his book is a good argument for the droplets-bouncing-on-an-oil-bath picture of quantum phenomena; if not then the rest of the book is really irrelevant. The book could have done with a lot less illustration of times when the scientific establishment was wrong. It's obviously possible they are all wrong, you just have to prove it.

[Laurence · Rating 5 out of 5]

V good read and fully recommend!

Like all the best pop science books, Escape from Shadow Physics excellently motivates the history, circumstances, and characters that have driven the relevant scientific advancements (and stagnations..) that have brought us to where we stand today. And fortunately, it’s super easy to read, which is v high praise considering it’s subject matter (or subject wave..? bad joke).

It’s been a long time since I read such a good pop science book. Perhaps Forrest Kay is a bit whiney at times, but I’m sold on his cause. Five stars!

[Lloyd Pierson · Rating 5 out of 5]

Excellent history, interesting theory

I really enjoyed the sections telling about the history of scientific ideas and the people who developed them. His own theory is cogently explained, backed up by experiments and analogies. It makes me believe that there will come a time when the reality that underlies the probabilities of quantum mecanics.

[Pratik Mehta · Rating 3 out of 5]

DNFed because it is such a fringe subject and very hard to carry on with knowledge our previous concepts still on mind. Finished 1/3rd of the book where author unravels classical vs quantum debates, provides historical erratics and explains how particle-wave debate evolved. Gave me a very strong foundation to basic understanding of physics.

[Rob · Rating 4 out of 5]

I expected this to be mostly about the pilot wave interpretation and maybe other alternatives to the Copenhagen interpretation, but that was only part of the book. More time was dedicated to history of science leading up to the present and how scientific theory develops. Very interesting.

[Ron Wade · Rating 5 out of 5]

Thoroughly researched, and thoroughly fascinating, argument for the hidden variables theory of quantum mechanics. Accessible to the layperson who has at least some familiarity with the history and development of the field. Highly recommended.

[Adam Orden · Rating 3 out of 5]

Very wonky,,,but I am into science and physics. The first half is much much better than the last half of the book. I you want to understand the history of quantum physics without being a mathmetician or physicist this is for youl

[Rory · Rating 5 out of 5]

Brilliant

[Cynthia Sprout]

Well beyond me

[Steven Kim · Rating 4 out of 5]

Dives deep into the history of quantum mechanics and all the people involved showing where things stand today. Really liked the narrative short chapters making things to the point and catchy.

[Michael Mitchell · Rating 2 out of 5]

A little disappointing. The author spent more time trashing supporters of the Copenhagen interpretation of QM instead of explaining HQA.

[Lewis]

Kind of want to move to Brazil and become a born again positivist and ironically worship at the church of Auguste Comte now I know that’s an option. (This is not the moral of the story).

[Hannah · Rating 5 out of 5]

"It is obvious that quantum mechanics is incomplete. It is a statistical theory, which means that it is incomplete as a matter of logic."