A Different Story Starts Here
The previous three parts were about building a model. From here on, the story is about a finished model producing an answer to our question — inference.
Keeping these two apart matters. Training changes the values inside the model; inference simply uses a model that has never changed. This is the point where practical conversations most often go wrong.
Let us follow, step by step, what happens inside from the moment you type into the chat window and hit send until the answer comes out one character at a time.
First It Becomes Numbers
The text you enter is split into tokens, as we saw in Part 1. And every token carries a unique number. If the vocabulary holds 200,000 entries, the numbers run from 0 to 200,000.
So the text first becomes a sequence of numbers. But numbers alone are not enough. There is no guarantee that number 17 and number 18 have similar meanings. A number is just a name tag.
That is why each number is converted into a bundle of numbers that carries meaning. This conversion is called embedding. After embedding, tokens with similar meanings sit numerically close to one another.
If the word embedding sounds familiar, it is probably because you already heard it in the context of RAG. It is the same concept as the embedding used when documents are turned into vectors and stored in a vector database. We will meet it again in the final part.
The Inference Engine Is Built in Many Layers
Once embedding is done, inference proper begins. The inference engine is a structure of many stacked layers.
The output of layer 1 goes into layer 2, that goes into layer 3, and so on through N layers. And what happens inside each layer is essentially the same.
Why repeat the same operation many times? Because one pass is not enough to grasp the relationships. The candidates get more refined with each layer. That is why more layers means more computation per inference.
The name for this structure is the transformer. An LLM is a thing that infers the next token by way of a transformer.
Attention — Measuring Relationships
The core operation inside a layer is attention. What it does fits in one sentence: it measures how related the tokens are to each other.
Here is an example. 「저희 집 고양이 이름은 로이예요. 품종은 벵갈이에요. 로이는 우리 집에 사는 고양이입니다.」 (Our cat is named Roy. The breed is Bengal. Roy is the cat that lives in our house.)
In the last sentence, 로이 (Roy) has a very strong relationship with 고양이 (cat) from the earlier sentence. It also has some relationship with 우리 집에 (in our house), but a weaker one. Attention computes all of this strength and weakness as numbers — for every token, against every token that came before it.
And if the computed values are left as they are, the features stay blurry. So a post-processing step runs once to make the prominent things more prominent. This post-processing part is called the MLP.
The attention computation itself also happens along several branches at once inside a layer. One such branch is called an attention head. So there are N layers, and inside each layer there are multiple heads. Repetition inside repetition, which makes the actual number of computations enormous.
Why This Approach Changed the Game
Before attention appeared, the mainstream approach in natural language processing was to read a sentence in order from the front. Process one word, carry the result forward, and move on to the next word.
This approach had a structural weakness: the earlier content grows fainter the further you go. It causes no trouble in short sentences, but in long text such as a novel, when a name introduced early is later referred to as 그것 (it), the reference was lost.
Instead of reading in order, attention lets every token refer directly to every other token. Whether a word is ten sentences back or immediately before, the way the relationship is measured is the same. It is not a structure where things are forgotten because of distance.
The idea was presented in the 2017 paper by Google researchers, Attention Is All You Need. Almost every LLM in use today branched off from it. At the time of publication, even the researchers themselves did not anticipate a ripple effect of this size.
Why It Pauses After You Hit Send
When you paste a long text into the chat window and send it, there is a brief pause before the answer starts coming out. After that, the characters follow quickly.
What happens during that first pause is the attention computation we just looked at. It is the stage where all the relationships across the entire input are measured, and this stretch is called prefill.
There is something easy to miss here. The computation does not cover only the text we typed. The invisible system prompt, the previous conversation history, and various configuration values all go in together. What you see on screen is not everything.
That is why this waiting time grows noticeably as the input gets longer. It is also why the first response gets slower as a conversation goes on — the earlier history is fed back into the computation every time.
The Candidates Get Scored
Once everything has passed through the layers, candidates for the next token come out along with scores. Since this is a model that has finished training, a car will not come out on top after 점심 메뉴는 (for lunch, the menu is). Names of foods line up at the top.
The values that come out at this point are not probabilities, just scores. They are numbers like 4.3, 1.7, 0.4, and negative values appear too. These raw scores are called logits.
They are awkward to work with as they are, so they are converted so that everything adds up to 1. Then each candidate can be read as a percentage. This conversion is called softmax.
This is how the next-token candidates and their individual probabilities are produced. Everything up to here is preparation for picking a single token.
The Model Does Not Change During Inference
There is something that must be pinned down here. In the inference process we have looked at so far, not a single parameter of the model changes.
The adjustment knobs from Part 2 stay fixed exactly as they were set during training. Inference is the act of flowing computation through that fixed configuration. Only training touches the knobs, and that has a completely different order of computational cost, so it cannot happen mid-conversation.
That is why the phenomenon of 어제 알려줬는데 오늘 또 모른다 (I told it yesterday and today it does not know again) occurs. What you told it in a conversation was only present in that conversation input; it was never engraved into the model. Open a new conversation and that input is gone.
The fact that the same question gets a different answer today than yesterday is also not because of training. We will look at the reason in the next part.
Not a Single Character Has Come Out Yet
We now have a candidate list and probabilities. But not a single character of the answer has come out yet, because what to actually pick from among them is still undecided.
It seems like always picking the top-ranked one would work, but that is not what happens. And that selection method is exactly what the values we adjust in the API — temperature, top-k, top-p — control.
The next part looks at that selection stage. That is where the reason lies for the answer changing a little each time for the same question.