The Previous Post Got Us to the Candidates
In the previous post, the input passed through every layer and produced next-token candidates each with its own probability. And yet not a single character of the answer has appeared.
One thing remains: which of those candidates to actually pick. This stage is called sampling.
It is a short stage, but its practical impact is large. Most of the values you adjust in an API hang on it, and so does the reason the same question gets a different answer every time.
The Top Candidate Is Not Always Chosen
It seems like always picking the highest-probability token would work fine. But that is not what happens.
Doing that would produce the exact same answer for the same input every time. It is predictable, but the writing turns monotonous, and once it heads down a wrong path it keeps circling the same spot.
So one candidate is drawn according to the probabilities. If first place is 60 percent, second is 25 percent, and third is 10 percent, first place usually comes out, but occasionally second does. This draw happens for every single token.
This is why the same question gets a slightly different answer each time. As stressed in the previous post, it is not because the model learned anything. The parameters are fixed; what changes is the outcome of the final draw.
temperature — How Wide to Leave the Candidate Pool
The value that sets the range of that draw is temperature. It is a temperature in the literal sense, and the intuition matches the name.
Raise the temperature and candidates further down the list get a chance to be drawn. Results become more varied, moving in the direction people commonly call creative. Lower the temperature and only the top candidates get drawn. Results stabilize, and repeated runs produce similar answers.
So is higher always better? No. Open it too wide and tokens that fall outside the context start getting in. After 「우리 점심 메뉴는」 (our lunch menu is), something other than food gets attached. Conversely, set it too low and sentences turn stiff and repeat the same phrasing.
The practical instinct is simple. Low for fixed-format output, classification, extraction, and code generation. High when you need several options, such as draft copy or a list of ideas. When in doubt, start on the low side and raise it as much as you need.
top-k and top-p — Two Ways of Cutting
Alongside temperature, the values you see most often are top-k and top-p. Both cut the candidate list, but they cut by different criteria.
top-k cuts by count. If k is 40, only the top 40 by probability remain and the rest are discarded. It is simple, but it cannot tell situations apart. It keeps 40 candidates where the answer is obvious and first place dominates, and it also keeps only 40 where the candidates are all comparable.
top-p cuts by cumulative probability. If p is 0.9, probabilities are added from the top until they reach 0.9, and only those remain. When first place dominates, only one or two remain; when the candidates are close, several remain. In effect, the width adjusts itself to the situation.
That is why top-p is the more common choice these days. There is no need to touch all three at once. Usually temperature alone is enough, and when the output keeps drifting, you tighten top-p along with it.
Context Narrows the Candidates in Advance
There is something people often miss here. The candidate list itself has already been shaped by the preceding context.
With only 「우리 점심 메뉴는」 (our lunch menu is), the candidates are broad. Anything edible can come next. But put 「집 앞에 일식집이 개업했다」 (a Japanese restaurant opened in front of our building) before it, and the top of the candidate list shifts to things like sushi, sashimi, and soba.
It is better to use this fact before touching temperature. When you have a direction in mind, giving clear context generally works better than tightening settings. The settings only operate within an already narrowed pool of candidates.
If advice to write good prompts has sounded vague, this picture makes it concrete. A prompt is the act of changing the shape of the candidate list.
Why the First Response Is Slow and the Rest Is Fast
One token has been drawn. But an answer is usually made of hundreds of tokens. To draw the next token, everything has to be computed again, including the token just drawn.
That makes it sound like the heavy computation from the previous post has to be repeated in full for every token. Done that way, a single character really would take a long time to appear.
Yet on screen there is only a brief pause at the start, after which characters follow quickly. That is because the results computed earlier are stored and reused. It is the same concept as caching in software development, and here it is called the KV cache.
So perceived speed splits into two segments. The prefill segment, which computes the entire input for the first time, is slower the longer the input; the segment after it, where characters come out one at a time, is relatively steady. When a response feels slow, which segment is at fault decides the remedy. If the front is slow, reduce the input; if the back is slow, reduce the output or use a faster model.
Stopping Is Also a Token
So when does the model stop? It is not that it decides on its own at some moment.
The vocabulary contains special tokens as well as visible characters, and among them is a stop token meaning 「여기서 끝」 (this is the end). When it is drawn as the next token, generation stops.
In other words, stopping is drawn by probability just like any other token. And where it is appropriate to stop is learned during the post-training process covered in part 3. That is when the sense that you should stop after answering the question gets attached.
Answers that cut off half-finished or run longer than necessary are easier to understand from this viewpoint. The remedy differs depending on whether the maximum output length limit was hit or the stop token was drawn early.
That Is Inference, End to End
The input is turned into tokens, embedded, passed through the layers to produce candidates, one is chosen by probability and appended, and it stops at the stop token. That is everything an LLM does to produce an answer.
Across five posts we have now covered both training and inference. What remains is translating this knowledge into practical judgment.
The final post answers the question posed at the very start of part 1. What determines whether a problem is solved with a prompt, with RAG, or by going all the way to fine-tuning?