What happens in the cockpit
An airliner's autopilot handles the cruise phase more steadily than a person. Holding altitude and heading finely through small shifts in wind is something a machine does far better. So what a pilot does during cruise is mostly watch the instruments.
That is where the problem starts. When a situation arises that the autopilot cannot handle, control passes to the human. But a situation the autopilot cannot handle is, by definition, a rare and complicated one.
So a person who has done nothing for hours is handed, within seconds and without warning, the hardest judgment the system just gave up on.
Lisanne Bainbridge set out this structure in 1983. The paper is called Ironies of Automation and the body runs five pages.
The first irony: designers are people too
The first irony the paper points at sits in an unexpected place.
The usual case for automation is that people are unreliable. People err, tire and behave inconsistently, so hand it to a machine. But the machine was designed by a person.
So errors in an automated system do not disappear; they move. Mistakes at the sharp end decrease, and in their place remain conditions nobody thought of at design time. Since those are exactly what people failed to foresee, they rarely surface in normal operation and go off precisely in rare situations.
Move it to AI and the same sentence holds. The argument that people are biased so we should hand it to a model is only half valid in front of the fact that the model was trained on data made by people.
Two layers of trap
The main point is what is left to the human. The paper identifies two layers.
First, what is left is not the easy work but the hard work. Automation eats the parts with clear rules and repetition first. What remains is exceptions, judgment and ambiguity. The human share does not shrink; it shrinks with only its difficulty raised.
Second, the chance to practice that hard work disappears, because the skill goes unused in day to day operation. Manual handling and situational feel both rust when unused. So at the moment it is actually needed, skill is lower than it was before automation.
A third layer stacks on. To monitor whether the automation is running correctly, you need to know whether its judgments are right. But the capacity to know that is precisely the capacity the automation replaced. To supervise the work you must be able to do it, and automation is why you no longer do it.
So the paper's conclusion comes out counterintuitive. Operators of automated systems need more training, not less, for the rare but decisive intervention.
Monitoring is what people are worst at
The representative job left after automation is monitoring, which happens to be the kind of work people are worst at.
This is not an impression but an old measurement. Around the Second World War, an experiment in Britain measured the performance of radar watchkeepers. Participants sat before a clock like device and had to catch the moments when the hand occasionally jumped two steps instead of one. Signals were rare.
The result was clear. About half an hour in, the miss rate rose noticeably. Not because people are lazy. The capacity to watch a screen where nothing happens for long stretches is simply something people have little of.
The conclusion is painful. Automation leaves the human the hardest judgments while simultaneously putting them on standby in the mode they are worst at. Both conditions land on the same person at once.
It has actually happened
An aviation accident over the Atlantic in 2009 shows this structure exactly.
When the airspeed instruments iced over and produced invalid readings, the autopilot disengaged itself. Control passed to the humans at that instant. Night, cruise altitude, part of the instrumentation untrustworthy, and a matter of seconds.
One of the points raised in the investigation was that this combination was something the crew had encountered almost nowhere, in training or in service. Flying manually at cruise altitude is itself rare, precisely because the autopilot works well.
The structure Bainbridge wrote out twenty six years earlier stands unchanged. The moment the machine gives up is the hardest moment, and practice for that moment does not happen in ordinary work.
The reason for citing this is not to fault aviation. Aviation is the field that has studied this problem longest and built the most countermeasures. The point is that it remains even so.
Automation bias: with an answer beside you, you look less
One more thing. Having automated advice beside you changes the checking behavior itself.
Researchers call the tendency automation bias, and it shows up in two directions. One is missing things. Problems the system did not flag go unnoticed by the person too. Places they should have checked themselves get skipped because the system was quiet.
The other is going along. When the system gives wrong advice, they follow it. Judgments they would not have made alone get made because an answer is on the screen.
The key is that this is not a lack of skill. With a plausible answer beside them, people automatically lower their checking intensity. They tilt toward saving effort, and because that is usually rational, it sets in.
This meets episode nine's conclusion. In front of smoothly written output, human checking weakens. And AI output is almost always smooth.
Forty years later, the same structure entered the office
This paper is read now because AI adoption has exactly the same shape.
Drafting is handed to AI. The human share becomes review. But review is not easier than drafting. Finding what is wrong in someone else's plausible prose is harder than writing from a blank page. The writer knows where they fudged; the reader receives text with those traces erased.
And the less often you draft yourself, the duller your sense of what is wrong becomes. The second trap operates unchanged.
Code is the same. Hand over the writing and what remains is reading and judging, which is harder than writing. Telling code that works from code that looks like it works is something only people who have written it can do.
The monitoring problem carries over too. In most cases AI output is fine. So review becomes the job of watching a screen where nothing happens. The job you start missing things on after thirty minutes.
This is not an argument against automating
Bainbridge's paper is not anti automation. The point is that when you design automation you must design the human's remaining share along with it.
Left alone, the human share is not designed but left over as residue. Whatever automation could not eat becomes the person's job. A share defined that way has never had its difficulty reviewed by anyone.
In practice it is hard to find the human side designed in an AI adoption plan. What gets automated is spelled out in detail, while what the remaining person does, and under what conditions, gets one line saying review and approval. That one line contains every trap in this episode.
The name for this state comes up in the next episode.
One term only: the ironies of automation
The ironies of automation are that as automation advances, the difficulty of the work left to people rises while the chance to practice that work falls.
The word irony is exact here because intent and result run opposite. An effort to lighten the human load leaves only the hardest part, and an effort to reduce error raises the chance of error at the decisive moment.
The badge on this episode is design principle for the same reason. This theory is not an ancestor of any AI algorithm. It governs how AI should be introduced. The countermeasures aviation and medicine built over decades transfer directly as reference material.
So what changes tomorrow
When handing work to AI, decide alongside it what you will keep practicing. Hand over everything and only review, and in six months your reviewing skill will have dropped too. There has to be one item a week you do from scratch yourself. This is not a preference but depreciation management.
Do not plan review as easier than drafting. If your schedule and staffing assume that review will be quick because AI writes the draft, that plan is already wrong. What is saved is writing time, not judging time.
Build the review checklist in advance. Reading it all and looking for something odd fails in front of smooth prose. Fix the check points ahead of time, figures, dates, quotations, conditions, exception handling, and sweep item by item. It is the same object as the verification list from episode six.
Redo a few items from scratch at random. This is the known way to hold performance on monitoring work. Reviewing everything is not sustainable; picking a sample and digging deep is. And that sample doubles as the practice that keeps your skill up.
Schedule practice for rarely used skills. This is why aviation uses simulators. If doing it from scratch without AI is not on the calendar, nobody does it.
In short: when deciding what to hand over, decide what to keep as well.