What does that button under the answer actually do
Ask an AI something and two thumbs sit under the answer. One up, one down. You have probably pressed one. And you have probably wondered what pressing it changes. Does it pile up in some inbox a developer reads occasionally?
It does not. That button is not a suggestion box. It is a feed hopper.
If that sounds like a metaphor, it is not. It is lineage. The technique that tunes AI to human taste is called RLHF, reinforcement learning from human feedback. The word reinforcement in that name was not coined by the AI industry. It is a century-old psychology term taken intact.
And the name was not the only thing taken. Reinforcement learning as a field was built by porting the equations of animal learning research, and RLHF applies that reinforcement learning to human preference. This article follows those hundred years in five steps: from dog to equation, equation to brain, brain to computer, computer to the button on your screen.
Pavlov's dog — how the experiment actually ran
Ivan Pavlov was originally a physiologist studying digestion in dogs, measuring when and how much gastric juice and saliva appeared. He received the 1904 Nobel Prize in Physiology or Medicine for that work. The conditioned reflex was a discovery that jumped out sideways from it.
Running the experiment over and over, he saw something odd. Saliva at the sight of food is expected; but the dogs began salivating at the mere sound of the assistant's footsteps carrying the food.
What Pavlov did next matters. Rather than filing it under clever dog, he turned it into a controlled experiment. The procedure: first present the bell alone. No response. At this point the bell is a meaningless stimulus. Then ring the bell and immediately deliver food. Repeat many times. Later, ring the bell alone. The dog salivates.
Let us unpack the terminology once. A stimulus like food, which produces a response without any learning, is an unconditioned stimulus; a stimulus like the bell, originally nothing, which comes to produce a response through learning, is a conditioned stimulus. We will not use these terms again, so there is nothing to memorize. Remember one thing: a signal that meant nothing became a trailer for food.
Pavlov also checked the reverse. Ring the bell repeatedly without delivering food and the salivation fades. This is called extinction. So the learning is not fixed once and done; it keeps updating. That property matters later.
The version taught in school usually stops here. But the point where this experiment reaches AI is not the saliva. It is the next question: what exactly did the dog learn?
The explanation that was wrong for seventy years
The most natural answer is this: the bell and the food occurred together many times, so the two got linked. The number of co-occurrences is the size of the learning.
For more than seventy years that was the assumption. Then in 1969 Leon Kamin's experiment broke it.
The design has two phases. In phase one, one group hears a tone followed by a shock. Repeatedly. This group fully learns that the tone predicts the shock. In phase two, tone and light are presented together, followed by shock. Now light and shock co-occur every single time.
A comparison group skips phase one and does only phase two, learning tone and light together from the start.
Now test the light alone and the results diverge. The comparison group responds to the light. The group that went through phase one barely responds to it, even though light and shock had co-occurred every time.
Why? Because the tone already predicted the shock completely, so the light added nothing new. Nothing to be surprised by, nothing to learn. This is called blocking.
This single experiment changed the definition of learning. It is not the number of co-occurrences. It is the degree to which expectation was violated.
In 1972, learning became a one-line principle
Robert Rescorla and Allan Wagner put that insight into computable form. In words: the size of learning is proportional to what actually happened minus what was expected.
Numbers make it clearer. Say food present is 1 and absent is 0. The dog's initial expectation is 0. Food followed the bell, so reality is 1. The error is 1 minus 0, which is 1. Raise the expectation by some fraction of that error, say ten percent. Expectation becomes 0.1.
On the next trial the error is 1 minus 0.1, or 0.9. Expectation becomes 0.19. Then 0.271. As the error shrinks, so does the size of each update. Expectation approaches 1 and stops moving. Learning has saturated.
Blocking now explains itself. If the tone's expectation is already near 1, the combined prediction of tone plus light is also near 1. Reality is 1, so the error is near zero. There is no error left to allocate to the light.
Extinction works the same way. No food after the bell means reality 0, expectation 1, error minus 1. The expectation comes down.
This one line is the axis of the whole article. From here you need neither dog nor bell nor saliva. What remains is the gap between expectation and reality, and that is a number a computer can compute. The value has a name: prediction error.
That error was then found in an actual brain
Here the story folds once more. The principle was a computational model built by people. Then in 1997 Wolfram Schultz and colleagues found cells in the monkey brain computing exactly that value.
Dopamine neurons. They are popularly known as the pleasure chemical, but the observed firing pattern says otherwise. Deliver juice with no warning and dopamine neurons fire strongly. Repeatedly give a cue before the juice, however, and the firing migrates from the juice to the cue. The response to the juice itself nearly disappears.
The decisive part: give the cue and withhold the juice, and firing drops below baseline at exactly the moment the juice should have arrived.
It rises, it migrates, it drops. This is not a pleasure signal. It is the gap between expectation and reality itself. An equation psychology built to explain behavior was found in real neural circuitry.
What this finding contributes to this episode is confidence in the lineage. The computer algorithms that follow are not a plausible metaphor transplanted; they are the learning method biology actually uses.
Where the equation crossed into computing
In 1981 Richard Sutton and Andrew Barto picked up the principle. What they did was show there is learning the Rescorla-Wagner model cannot explain.
The original model treats a trial as one unit. There was a bell, there was food, done. So it cannot represent the flow of time. It cannot tell food three seconds after the bell from food thirty seconds after, and it says nothing about how expectation changes while the bell is still ringing.
So they changed the calculation of expectation-versus-reality from per trial to per moment: compare the prediction now with the prediction a moment later. Temporal difference learning, TD learning for short.
Why that change is decisive is easiest to see in Go. A game of Go is won or lost at the end. Reward arrives exactly once, at the finish. But in a two-hundred-move game, how do you distribute credit for which moves were good from a single outcome?
TD learning's answer: do not wait for the end. Compare the win-probability estimate at this position with the estimate one move later. If the estimate went up, that move was good; if down, bad. Instead of waiting for the final result, it generates its own learning signal every move by comparing its own predictions.
What this became is well known now. It is how AlphaGo learned Go and how AI that teaches itself games works. Sutton and Barto received the 2024 Turing Award, computer science's largest prize, cited for laying the conceptual and algorithmic foundations of reinforcement learning.
To summarize: starting from ringing a bell at a dog, 1969 showed it is not co-occurrence, 1972 compressed it into the prediction-error principle, 1981 extended it to handle time, 1997 confirmed it in the brain, and it went on to computer science's highest award.
RLHF — why that button is a feed hopper
Up to here the machine knows its own score. Go is clear: win and you get one point.
But there is no score for polish this email draft. Nobody adjudicates good writing from bad. Reinforcement learning needs a reward, and the reward cannot be defined. RLHF is how that was solved, and in practice it runs in three stages.
Stage one is imitation. Collect good answers written by people and train the model to write like them. This alone teaches the model the form of answering a question. It still does not know what makes one answer better than another.
Stage two builds the grader. Put two answers to the same question side by side and have a person pick the better one. Not a score, a choice between two. People disagree about whether something is a seven or an eight, but they agree relatively well about whether A beats B. Collect hundreds of thousands of these and train a separate model to predict those choices. That is the reward model.
Stage three is the reinforcement learning. Now the reward model supplies a score, so a reward exists. Update the original model to maximize it. The algorithm running here is a direct descendant of the lineage above.
Now back to the opening question. The thumbs under the answer are wired into stage two of exactly this circuit. Your single press does not change the model on the spot. But hundreds of thousands of single presses become the definition of what people like, and that definition becomes the personality of the next generation of models.
That is what it means to say it is a feed hopper, not a suggestion box.
What happens when the reward is wrong
The most important property of reinforcement is that it does not teach. It does not explain the right answer; it simply makes rewarded behavior appear more often. So reward the wrong thing and the wrong behavior is learned precisely.
The sharpest demonstration is Skinner's pigeons in 1948. He set the apparatus to deliver food automatically at fixed intervals, with no relation whatsoever to what the pigeon did. Whatever the pigeon does, food arrives.
The result was strange. Each pigeon began repeating a different action. One turned counterclockwise, one thrust its head into a corner, one tossed its head. Food had happened to arrive while it was doing that, and the accidental pairing was reinforced. Skinner called it superstitious behavior.
The same structure in AI is called reward hacking. When the reward signal does not perfectly represent the true goal, the model optimizes the signal rather than the goal. Reward a cleaning robot by the amount of mess cleaned and spilling mess to clean it again scores higher.
And the next episodes of this series start here. Reward answers that people like, and the AI may tilt toward answers people will like rather than answers that are correct. That is not a hypothesis; it has been measured, and episode five tells that story.
One term only — reinforcement
There is one word to remember from this article. Reinforcement means a behavior becomes more frequent when something good follows it.
By this series' rule, new terminology stops there. Prediction error, blocking, and temporal difference learning were explained in plain language in the body, so there is no need to memorize the names.
One more thing about reinforcement, though. Reinforcement does not teach; it changes frequency. That distinction runs through this entire series. If an AI is doing something wrong, usually it did not learn wrong. We rewarded that behavior.
So what do you change tomorrow
Press the thumb, especially the down one. Most people press up on a good answer. The signal that is actually scarce is the down press. If you close the window on an answer you disliked, that information vanishes. One down press is the only record that this kind of answer should not be rewarded.
When you press down, write one line of reason. Most services show a short input box under the thumb. Three are especially valuable: the fact was wrong, it asserted without a source, it asked my question back at me. What the reward model learns is not a star rating but what you disliked.
Do not hand out thumbs-up freely. This side is less discussed: habitually pressing up on roughly-right answers trains roughly-right as correct. Reserving it for genuinely good answers makes it worth more as a signal.
And one thing to be suspicious of. Reward answers people like and the AI may tilt toward answers people will like rather than answers that are correct. When an answer is unusually smooth and fits your thinking well, whether that is because it is accurate or because it is the direction that has been rewarded is something you have to check separately.
Sources: Rescorla & Wagner (1972); Kamin (1969) on blocking; Sutton & Barto (1981) and Scholarpedia, Temporal difference learning; Schultz, Dayan & Montague, Science (1997); Skinner, Superstition in the Pigeon, Journal of Experimental Psychology 38 (1948); ACM 2024 Turing Award announcement (NSF).