If you only know tokens as a billing unit
API price lists quote a rate per million tokens. Context limits are stated as 200K or 1M tokens. It is easy to file tokens away as 'something like character count, used for billing.'
But tokens were not invented for billing. They emerged out of necessity, before training even begins, in the step that turns raw text into material a machine can work with. Billing merely borrowed the result.
Understanding that order explains three things practitioners hit constantly: why Korean costs more than English, why the same text yields different token counts on different models, and why converting a context limit into a character count is always slightly wrong.
This article follows that process from the beginning. No prior knowledge is required.
It all starts with a corpus
The word corpus comes first. It sounds technical, but the meaning is plain: a body of text. A great deal of collected writing.
Text is scattered everywhere on the internet. It sits in blogs, on homepages, across social media. Code on GitHub is written characters too, so it counts as text. Even limiting the collection to public pages that need no login yields an amount that is hard to picture.
Stitch all of that into one enormous block of text and you have a corpus. That is the starting material for a model.
It helps to fix the destination first. The goal is a machine that predicts what comes after a fragment like 'our lunch menu is.' A machine that looks at what came before and predicts what comes next. Fundamentally, that is the one thing an LLM does.
Data and a dataset are not the same thing
No matter how much text you gather, you cannot train on it as-is. Training requires material organized to a specification, and that organized form is called a dataset. Data and dataset are different words.
So a preprocessing step comes first, turning a corpus into training material. Tokens are the output of that step.
The order is: corpus (just long text) → preprocessing → tokens (training material). What follows examines what happens inside that middle arrow. It splits into two stages.
Stage one — turn characters into numbers
Computers do not know characters. They handle only numbers. So the first step converts characters to numbers, and the rule for doing so is called an encoding.
You have probably met the word already. It is the encoding that turns up when you download a file, the text renders as garbage, and you go searching for why. The most widely used scheme today is UTF-8.
In UTF-8 a single Korean character is stored as three bytes. One byte is eight positions of 0 or 1, and eight positions give 256 possibilities. Counting from zero, that is 0 through 255. The reason 255 shows up so often in network settings is the same arithmetic.
Apply this rule across the whole corpus and every character in the world — Korean, English, emoji, obscure symbols — becomes a number. The entire body of text turns into one long run of numbers. That run is called a sequence.
That is stage one. No tokens yet.
Stage two — merge repeated fragments to compress
Using the raw number run is wasteful, so it gets compressed.
Image compression is a useful analogy. Shrinking an image to JPEG means finding repeated patterns, recording each once, and replacing the rest with references. Text compression uses the same idea.
Scan a long body of text and the same combinations recur endlessly. In English it is fragments like 'ing' and 'the'; in Korean it is polite verb endings. Find those frequently adjacent fragments and merge each into a single entry. The resulting list of entries is the vocabulary.
An algorithm decides what to merge. The most widely used is BPE (Byte Pair Encoding), which repeatedly fuses the most frequently adjacent pair of fragments for a fixed number of rounds. Each entry in the vocabulary that results is a token.
So the definition lands here. A token is not a word. It is one entry in a vocabulary produced by a compression algorithm.
Counting it yourself makes it obvious
Description stays abstract, so here are actual measurements, taken with the o200k_base tokenizer used by the GPT-4o family.
The Korean word for lunch is a single word but splits into two tokens. The phrase 'our lunch menu is' looks like three chunks to the eye but counts as five tokens. You can also see that a leading space travels attached to the fragment that follows it.
This makes clear that a token is neither a word nor a character. Sometimes it is finer than a word; sometimes several words fuse into one. The only criterion is how often those pieces sat next to each other in the training corpus.
'점심' (lunch) → 2 tokens ['점', '심']
'우리 점심 메뉴는' → 5 tokens ['우리', ' 점', '심', ' 메뉴', '는']
'Hello world' → 2 tokens ['Hello', ' world']The same sentence: 43 tokens, 21 tokens, 10 tokens
A vocabulary is built from the corpus during training. Different model, different vocabulary, different token count for the same sentence.
I measured one Korean sentence across three tokenizers. On p50k_base, from the GPT-3 era, it comes to 43 tokens. On cl100k_base, used by GPT-4, 21 tokens. On o200k_base, used by GPT-4o, 10 tokens. The same sentence, a more than fourfold spread.
Meanwhile the English sentence 'What should we have for lunch?' is 7 tokens on all three. The divergence appears only in Korean.
The cause lies in the size and makeup of the vocabulary. Early tokenizers were built around English and held almost no Korean fragments, so Korean shattered into raw UTF-8 bytes. Decompose the Korean word for lunch with cl100k_base and it breaks mid-byte rather than at character boundaries, producing mangled pieces. Later models grew their vocabularies and absorbed whole Korean fragments, which cut the count.
This is where the claim that Korean costs more than English comes from. What matters in practice is that this is not a fixed fact but a value that moves with your choice of model.
How to check for yourself
Counting once beats reading an explanation. A browser is all you need.
Open tiktokenizer.vercel.app. Type any sentence into the left pane. Tokens appear color-coded on the right with a total underneath. Change the model selector at the top and you immediately see how the count shifts for the same sentence.
Three things are worth trying. First, put a sentence you use often next to its translation and compare counts. Second, hold the sentence fixed and cycle through models. Third, enter uncommon proper nouns such as company or product names. You will see that the less common a term is, the more finely it fractures.
The third is the most useful. If your prompt structure repeats proper nouns, that is a likely place where tokens are being consumed faster than you expected.
Where this concept bites in practice
First, cost estimation. Converting character counts into tokens generally misses for languages other than English. Measuring a representative prompt with the tokenizer of the model you will actually use is accurate.
Second, context limits. How many characters fit in 200K tokens depends on the language and the model. Knowing your headroom means measuring it.
Third, switching models. If you moved to a cheaper model and costs did not fall as much as expected, the unit price may have dropped while the tokenizer split your text more finely, raising the count. Real cost is unit price and token count together.
All three are answerable by measurement, and measuring takes under five minutes.
That is the material prepared
We gathered a corpus, converted it to numbers, and merged repetitions into tokens. The material for training is ready.
The next installment looks at how a neural network actually learns from this material: hiding the answer, asking it to guess, and correcting backward in proportion to the error. What gets corrected is the parameters. That is also where numbers like 7B and 70B beside a model name will finally make sense.