Emre Sevinç

“Yann LeCun's strategy provides a good example of a much more general notion: the exploitation of innate knowledge. Convolutional neural networks learn better and faster than other types of neural networks because they do not learn everything. They incorporate, in their very architecture, a strong hypothesis: what I learn in one place can be generalized everywhere else.

The main problem with image recognition is invariance: I have to recognize an object, whatever its position and size, even if it moves to the right or left, farther or closer. It is a challenge, but it is also a very strong constraint: I can expect the very same clues to help me recognize a face anywhere in space. By replicating the same algorithm everywhere, convolutional networks effectively exploit this constraint: they integrate it into their very structure. Innately, prior to any learning, the system already “knows” this key property of the visual world. It does not learn invariance, but assumes it a priori and uses it to reduce the learning space-clever indeed!”
― Stanislas Dehaene, How We Learn: Why Brains Learn Better Than Any Machine . . . for Now

“The moral here is that nature and nurture should not be opposed. Pure learning, in the absence of any innate constraints, simply does not exist. Any learning algorithm contains, in one way or another, a set of assumptions about the domain to be learned. Rather than trying to learn everything from scratch, it is much more effective to rely on prior assumptions that clearly delineate the basic laws of the domain that must be explored, and integrate these laws into the very architecture of the system. The more innate assumptions there are, the faster learning is (provided, of course, that these assumptions are correct!). This is universally true. It would be wrong, for example, to think that the AlphaGo Zero software, which trained itself in Go by playing against itself, started from nothing: its initial representation included, among other things, knowledge of the topography and symmetries of the game, which divided the search space by a factor of eight.

Our brain too is molded with assumptions of all kinds. Shortly, we will see that, at birth, babies' brains are already organized and knowledgeable. They know, implicitly, that the world is made of things that move only when pushed, without ever interpenetrating each other (solid objects)—and also that it contains much stranger entities that speak and move by themselves (people). No need to learn these laws: since they are true everywhere humans live, our genome hardwires them into the brain, thus constraining and speeding up learning. Babies do not have to learn everything about the world: their brains are full of innate constraints, and only the specific parameters that vary unpredictably (such as face shape, eye color, tone of voice, and individual tastes of the people around them) remain to be acquired.”
― Stanislas Dehaene, How We Learn: Why Brains Learn Better Than Any Machine . . . for Now

“Note that in the above construction we made a number of choices; here we must beware. Choosing a good categorification – like designing a good algebraic structure such as that of preorders or quantales – is part of the art of mathematics. There is no prescribed way to categorify, and the success of a chosen categorification is rather empirical: its richer structure should allow us more insights into the subject we want to model.”
― Brendan Fong, Seven Sketches in Compositionality: An Invitation to Applied Category Theory

“Our brain is therefore not simply passively subjected to sensory inputs. From the get-go, it already possesses a set of abstract hypotheses, an accumulated wisdom that emerged through the sift of Darwinian evolution and which it now projects onto the outside world. Not all scientists agree with this idea, but I consider it a central point: the naive empiricist philosophy underlying many of today's artificial neural networks is wrong. It is simply not true that we are born with completely disorganized circuits devoid of any knowledge, which later receive the imprint of their environment. Learning, in man and machine, always starts from a set of a priori hypotheses, which are projected onto the incoming data, and from which the system selects those that are best suited to the current environment. As Jean-Pierre Changeux stated in his best-selling book Neuronal Man (1985), “To learn is to eliminate.”
― Stanislas Dehaene, How We Learn: Why Brains Learn Better Than Any Machine . . . for Now