Why recognizing a face is hard
Recognizing a face in a photo is so easy for people that the difficulty is invisible. To a computer, though, a photo is just an array of numbers. A thousand-by-thousand greyscale image is a list of a million brightness values, and the computer must take that list and answer: this is a cat.
The naive method is to memorize the cat photo whole: remember the pattern of a million numbers and compare new photos against it. That fails immediately. Move the cat ten pixels sideways and all million values shift to different positions, producing a completely different list. To a human eye it is the same cat; as numbers they are strangers.
The same happens if brightness changes slightly, or if the cat tilts. So memorization would require storing every combination of position, brightness, and angle for a single cat, and the number of combinations is effectively infinite.
So what you need is a way to see the same thing even when position shifts a little. Some device that finds a fragment but forgets exactly where it was. The original of that device was inside a cat's brain.
1959, an anesthetized cat and one electrode
David Hubel and Torsten Wiesel inserted a microelectrode into a cat's primary visual cortex, an apparatus that picks up the electrical signal a single brain cell makes when it fires and converts it to sound. They then projected patterns on a screen before the anesthetized cat's eyes, hunting for which images made which cells respond.
For a while there was nothing. Dots, circles: the cell stayed quiet. Then, as they swapped a slide, the speaker erupted with a cell firing hard. It was not the picture but the edge of the shadow as the slide slid in, a single tilted line segment, passing through.
Chasing that accident down revealed two kinds of cell.
Simple cells respond only to a line at a particular angle. A cell tuned to 45 degrees is nearly silent at 90. They also care where the line falls: each cell owns a patch of the visual field and fires only when its angle appears in its patch.
Complex cells care about angle the same way but do not care about position. Given a 45-degree line they respond whether it sits a little higher or lower. The 1962 follow-up worked out these complex cells and the orientation columns, where cells tuned to the same angle cluster in a column.
The problem from the previous section is solved right here. Simple cells find the fragment; complex cells let its position slide. A device that finds a fragment but forgets where it was had already been wired, in two layers, inside a brain.
The two received the 1981 Nobel Prize in Physiology or Medicine, sharing half of it between them alongside Roger Sperry, cited for their discoveries concerning information processing in the visual system.
From parts to whole — the idea of stacking layers
What Hubel and Wiesel left behind was not only two cell types. More important was the relationship between them. A complex cell looked like something assembled from the outputs of several simple cells. Bundle simple cells tuned to the same angle but watching slightly different patches, and you get a cell that responds to that angle without caring about position.
So what happens if you do the bundling once more? Bundle short segments into corners, corners into curves, curves into an eye shape, and eyes and nose and mouth into a face.
That picture is this episode's blueprint. Alternate a layer that finds simple things with a layer that lets position slide, and the higher you go the larger and more complex the things recognized, and the less sensitive to position.
1980, those two cells became code
The Neocognitron built by Kunihiko Fukushima is not a metaphor. In the paper he cites Hubel and Wiesel's findings directly and names simple cells the S layer and complex cells the C layer, after simple and complex.
The structure alternates S and C layers. The lower S layer finds short segments, the C layer above blurs position, the S layer above that finds a larger fragment in the blurred result, and the C layer above blurs position again. Repeat for several tiers.
In other words: find parts, let position slide, find bigger parts, slide again. Write what a CNN does today in one sentence and this is it.
The Neocognitron recognized handwritten digits despite small shifts in position. Its learning method differed from today's, though: it found patterns on its own without being told the right answers, which made it hard to push performance up. That last piece was filled in nine years later.
1989, it became practical reading postal codes
Yann LeCun and colleagues at Bell Labs built a network that reads handwritten ZIP codes, trained on digit images taken from mail actually collected by the US Postal Service. For its time it was a rare case of a neural network attached to a real operational problem.
What matters about this paper here is not performance but the reference list. It cites Fukushima's 1980 Neocognitron directly and states that its architecture is reminiscent of it but uses backpropagation instead of unsupervised learning. The same backpropagation from the previous episode.
So the lineage runs through documents, not circumstance. Cat visual cortex (1959), then Neocognitron (1980), then CNNs with backpropagation (1989). The widely known LeNet-5 comes in 1998, and development continues after that.
One more idea got attached here. Finding a line at a given angle is the same computation whether you do it in the top-left or bottom-right of the image. So rather than learning separately for each position, apply one computation rule across the whole image, sliding it along. This is weight sharing, and the sliding operation is called convolution. That is the C in CNN.
That single idea decided practicality. Learning separately per position means millions of things to learn; sliding one rule means hundreds. Less data, lighter computation.
One term only — receptive field
A receptive field is the narrow patch of the visual field that a single neuron is responsible for.
The cell Hubel and Wiesel probed did not look at the whole screen. Each sees only a very small patch and knows nothing about what happens outside it. That is why a CNN does not take in a photo at once but sweeps a small window across it.
Receptive fields widen as layers rise. A higher-layer cell bundling several lower-layer cells covers a correspondingly wider patch. That is why lower layers see line segments and higher layers see faces.
One important property follows. This structure assembles from parts to whole. It finds fragments well, but the situation and context those fragments compose is, in principle, a different problem. That property produces the practical guidance in the next section.
So what can this structure not do
The limits of assembly-from-parts have surfaced repeatedly.
First, if all the parts are present it will say yes even when the arrangement is wrong. A face with eyes, nose, and mouth all present but scrambled in position can still be judged a face. It sees the presence of parts well and the relations among them weakly.
Second, changes invisible to the human eye can flip a judgment. Nudge pixel values minutely and a panda becomes a gibbon. A person cannot even see the difference between the two images.
Third, it uses the background as evidence. Train on cows only in pastures and it fails on a cow at the beach. It learned green background, not cow shape, as the evidence for cow.
These three are not bugs. They follow from what the structure is. And they translate directly into practice.
So what do you change tomorrow
Ask image AI what is visible, and verify what the situation is. Fragment detection is this structure's day job and it never tires the way a person does. The meaning of a scene, by contrast, is an assembled result, so when it is wrong it is wrong plausibly. Is there a hard hat in this photo can be delegated; is this work safe is looked at by a human.
Cut the question small. Rather than having one photo interpreted whole, narrow it: is there text in this region, is there a flaw on this part. Accuracy rises noticeably. A structure with narrow receptive fields does best when the question is narrow too.
Ask twice with angle and brightness changed. Position shifts are absorbed by the complex-cell layer; rotation and lighting are not. If rotating the photo or adjusting brightness changes the answer, that judgment is not yet trustworthy.
Deliberately mix in samples with different backgrounds. When building a validation set, include the same object photographed in a different place under different lighting. Whether it learned the background as evidence shows up only in those samples.
Sources: Hubel & Wiesel (1959, 1962); Nobel Prize in Physiology or Medicine 1981; Fukushima, Neocognitron (1980); LeCun et al., Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation 1(4) (1989).