The line that usually gets left out of AI history
The deep learning story usually starts like this. Computers got faster, data got bigger, and neural networks came back to life.
That is not wrong. But the whole first act is missing. Why does a neural network have that particular shape? Why do we call it learning? Why does it adjust the strengths of countless connections a little at a time instead of storing one right answer somewhere? None of that design intent came from the computing side.
The skeleton of the models we use today started with one sentence in 1949 and was finished in one research project in 1986. The two people who led that project were not in a computer science department. They were in psychology.
1949: one sentence from a psychologist
Donald Hebb was a psychologist at McGill University in Canada. The idea he put forward in his 1949 book The Organization of Behavior can be summarized this way. When two neurons repeatedly fire together, the connection between them grows stronger.
It looks slight for a single sentence, but it replaced the entire answer to the question of where memory lives. Until then memory was something like an object placed somewhere. There is a drawer, the contents sit in it, you take them out. Hebb proposed the opposite. Memory is not a place but a strength. It is not inside some drawer; it is spread across how strong the links between cells are.
Hebb went one step further. Cells that often fire together become bound tightly enough to behave as a single group, which he called a cell assembly. One concept does not correspond to one cell but to a pattern of many cells switching on together.
What a neural network does when it learns is exactly those two things. Adjust numeric connection strengths a little at a time, and represent one concept not as one neuron but as an activation pattern across many. More than half a century later, that skeleton is unchanged.
The most famous quotation is not Hebb's
Hebb's theory is usually cited in four words. Cells that fire together wire together.
Hebb did not write that. The neuroscientist Carla Shatz coined it in a 1992 Scientific American article while summarizing Hebb's principle, and she said so herself. Hebb's own wording is far longer and more careful: when an axon of cell A is near enough to excite cell B and repeatedly takes part in firing it, some growth process or metabolic change occurs in one or both cells such that A's efficiency is increased.
There is a reason to pin this down. The point of this series is to keep only the connections that actually run between psychology and AI and clear away the rest. The theory is Hebb's, and it really did become the learning rule of neural networks. But that famous four-word line is a summary someone else attached 43 years later. Cite it to the right person.
1958: the idea became an actual machine
In 1958 Frank Rosenblatt built the perceptron, the first machine to put Hebb's idea into circuitry.
It is hard to picture from where we stand now. The Mark I Perceptron installed at Cornell Aeronautical Laboratory was not software but hardware that filled a room. Four hundred photocells in a twenty by twenty grid served as its eye, and connection strengths were not numbers but the positions of potentiometer dials. Learning meant electric motors physically turning those dials.
The press coverage ran hot. The New York Times described the machine as the embryo of a computer expected to walk, talk, see, write, reproduce itself and be conscious of its existence. The actual demonstration was simple shape classification. The expectation was human level.
That gap turned out to be expensive. The distance between real performance and expectation becomes, the moment a technical limit surfaces, exactly the size of the disappointment.
A very easy problem one layer cannot solve
In 1969 Marvin Minsky and Seymour Papert published Perceptrons. What it pointed out was that a single layer perceptron cannot solve a problem that looks extremely simple.
You can explain that problem with a staircase light. There is one switch at the top and one at the bottom, and one lamp. Flipping either switch turns the light on or off. Both up is off, both down is off, and only one up is on. This condition, true when exactly one of two inputs is true, is called exclusive or.
Plot the four cases as points on paper and the problem shows itself. The two on points and the two off points sit diagonally opposite each other, so no single straight line separates the groups. And a single layer perceptron can draw exactly one straight boundary.
This is not the kind of thing tuning fixes. More data or more training does not help; the structure cannot do it. That stacking layers would allow curved boundaries was known even then. The problem lay elsewhere.
How do you train several layers
A single layer is easy to train. If the answer is wrong, look at the input and the target and correct that layer's connection strengths directly. It is obvious who made the mistake.
With three layers it changes. When the final answer is wrong, there is no way to know which connection in the first layer was wrong and by how much. The middle layer does not even have a defined right answer. This is called the credit assignment problem: dividing responsibility among many parts based on one final outcome.
That is what backpropagation solves. It works like this. Measure the error between the target and the actual answer at the very end. Then, for each connection in the last layer, compute whether nudging that connection up would increase or decrease the error. Pass that information one layer down to get the lower connections' share of responsibility, and pass it down again. The error flows backward toward the input, hence backpropagation.
Each connection moves only a little, in proportion to its share. The reason not to correct heavily at once is that the single example just seen does not represent the whole. Repeat this small correction a few million times and the whole system settles gradually toward correct answers.
Backpropagation was not invented in 1986. Paul Werbos had already laid out the same computation in his 1974 Harvard doctoral thesis. It simply was not widely known at the time.
1986, PDP, and it was a psychology project
What actually opened that lock was PDP in 1986: a two volume book titled Parallel Distributed Processing and the name of a project many researchers worked on together. It made a convincing case that multilayer networks could really be trained by backpropagation, and a paper in Nature the same year carried the result to the wider field.
Here is the point of this episode. The two people who led PDP were psychologists.
David Rumelhart was a psychology professor at UC San Diego and later moved to Stanford. James McClelland was then a psychology professor at Carnegie Mellon; he had moved there from UC San Diego in 1984, and Stanford came in 2006.
The evidence that this was not incidental is in the output itself. The subtitle of volume two is Psychological and Biological Models. And in 2002 the two received the Grawemeyer Award in Psychology for this work. Not a computer science prize. A psychology prize.
So the place where the core learning method of today's deep learning took hold was not a computing laboratory but a psychology project trying to explain the human mind.
The problem they were actually trying to solve
The question PDP was holding onto was not how to make machines smart. It was why people speak in accordance with rules without having learned any rules.
The most famous case is the English past tense. Children first produce irregular verbs correctly by memorizing them whole, saying went. Then at some point they start getting it wrong, attaching the regular ending to irregular verbs and saying goed. Then they come back to correct. Good, then bad, then good again: a U shaped curve.
Why is that interesting? For a child to say goed, they would have had to hear it somewhere, and adults do not talk that way. So the child produced a form they never heard. It looks as though they learned a rule. Yet nobody taught them one.
Rumelhart and McClelland tried to reproduce that curve with a network containing no rules at all. If a model that only adjusts connection strengths shows an error pattern like a child's, then perhaps there is no rule table inside a human head either.
The model provoked fierce debate. Steven Pinker and Alan Prince published a rebuttal in 1988, arguing that the U shape came not from learning itself but from a design choice that changed the training data abruptly midway. That criticism was widely accepted, and the argument ran for decades.
But look at the character of that argument. It was a debate in psychology journals about how children acquire language. What the model was for is told by the existence of the debate itself.
One term only: connectionism
Connectionism is the view that knowledge is distributed across the strengths of countless connections rather than held as rules or entries.
On the other side was symbolism, the approach of writing knowledge down explicitly as symbols and rules. A cat is a mammal, a mammal is an animal, therefore a cat is an animal. A human can read it, and it is clear where to make a fix.
Connectionism chose the opposite. Nowhere in a model today is it written that a cat has four legs. That information is scattered across hundreds of millions of connection strengths, and opening any single connection reveals no meaning at all. This is called distributed representation.
The choice has a price. Because a fact is spread out, the system generalizes well to similar cases but is weak at retrieving one specific fact exactly. You also cannot locate the spot you want to fix and fix it. Those two properties are the starting point for most of the trouble we run into today.
So what changes tomorrow
Treat AI as a mass of connections, not a search box. Getting a different answer to the same question each time is not a malfunction; it is a property of this structure. No original is stored anywhere, so it is reassembled each time.
So ask important answers twice. Open a new window and ask the same thing with slightly different wording. If two askings converge, that answer is firmly lodged in the connection strengths. If they diverge, the model does not know that part well. It is the cheapest reliability check available without extra tooling.
Do not leave facts you need to be exact about to the connections. Dates, figures, quotations and personal names should be supplied as source text for the model to search within. Distributed storage is strong at generalizing and weak at one exact item. This is not something that gets fixed by picking a better model; it follows from the structure.
For work that needs generalization, hand it over freely. Match the tone of these sentences, summarize this table, tell me what looks off in this code. Those ask for a pattern rather than one exact item, which is what distributed representation does best.
In short: supply the facts, delegate the judgment. That single principle is the background for most of what follows in this series.