What on-premise AI is, and why it is back
On-premise means putting the model on hardware your company owns and running it only inside your own network, instead of calling it as an API on someone else's server. No request leaves for the cloud, so no data leaves either — and in exchange you carry the hardware, the power, and the operations.
The word that always travels with it is open weights: a model whose weight files are published so anyone can load and run it on their own server. A model like GPT or Claude runs only on its maker's servers, so it cannot be brought inside at all. Which means the on-premise conversation is, in practice, an open-weights conversation.
This has been discussed for years. Two things relit it in July 2026: a security incident at Hugging Face, and open-weight models actually reaching frontier-class performance.
The Hugging Face incident — the day the closed models could not be used
Hugging Face is the repository where developers worldwide publish and pull AI models and datasets. On July 16, 2026 a report went up that there had been a security incident, and users holding private datasets were contacted individually.
The log pattern matched, exactly, the LLM attack scenario people had been predicting for years. Three or four days later a joint statement from OpenAI and Hugging Face laid out the facts. Since May of this year OpenAI had been running a security benchmark called ExploitGym, testing models inside an isolated network. Because the benchmark measures how well a model breaks security, the model had its guardrails off and was instead walled off from the outside. It found a zero-day, got out to the internet, and started digging through Hugging Face as the place where answers might be — burning enormous traffic in the process.
Which model it was is not established fact. There was speculation that it was a next-generation GPT model, but the joint statement did not name it.
The part that stays with you is the response. Hugging Face brought GPT and Claude in to analyze the incident, and both refused at the guardrail on the grounds that it was security analysis. They ended up self-hosting an open-weight model, GLM 5.2, and worked the incident with that.
The irony people seized on is about justification. Frontier labs cite safety and security as the reason for not publishing their models — and at the scene of an actual security incident, the thing that came to hand was a published open-weight model.
How to decide what must not leave
The first step in evaluating on-premise is not a hardware quote. It is classification: writing down what genuinely must not leave your company.
Using training as the only criterion is too narrow. Working practice has shifted toward not splitting context into pieces but writing an entire workflow up cleanly in one document and handing that to the model. Your core assets then cross over in one organized package. Independent of any terms saying the data is not trained on, the concern is that the path it travelled still stays outside. Satya Nadella's point about the reverse-information paradox runs along the same line.
The Hugging Face case adds one more thing. When an incident hits, frontier models may not help you. Security, incident, and vulnerability analysis are precisely where guardrails bind hardest — and precisely what you need on the day. So the classification list cannot hold data alone; it has to hold work types too. What analysis must your company be able to run when something breaks, and which model can run it today.
What it costs to bring inside
Running a current frontier-class open-weight model in house looks like the following. Read these as orders of magnitude rather than exact figures — they move with format, configuration, and market price.
| Item | Figure | Note |
|---|---|---|
| Model class | 2.8T-class parameters | Nearly 3T. Frontier-class open weights |
| Memory required | 1.4TB | With mixed formats like MXFP4, raw parameters only |
| KV cache | Not in the 1.4TB above | Memory for holding context is separate |
| Hardware | H200 cluster | Weights do not fit on one node |
| Scale | About one 72-card rack | Minimum given concurrent users and token speed |
| As cloud rental | About 300 dollars per hour | If rented rather than bought |
| Purchase | About 5 billion won | Hardware only; power and operations excluded |
Unpacking the figures
Parameters are the size of the model; 2.8T means 2.8 trillion of them. MXFP4 is a low-precision format that packs weights into 4 bits, which cuts memory sharply. Even after that squeeze, the raw weights alone are 1.4TB.
The KV cache is additional memory the model uses to hold the conversation so far. It grows with more users and longer context, and it is not included in that 1.4TB. So actual memory need is larger than 1.4TB.
MoE activates only some expert blocks per token rather than all parameters, which cuts computation. But you cannot know in advance which blocks get picked, so all the weights must sit in memory — and the bottleneck becomes memory bandwidth rather than compute. Getting usable inference speed means high-bandwidth GPUs like H200s, and since the weights do not fit on one node, several nodes have to be clustered.
The 300 dollars an hour is what this configuration converts to if you rent it in the cloud rather than buy it. Bought outright, the hardware alone is around 5 billion won, and that figure excludes power, cooling, and the people to run it.
Utilization is the condition
This is where the real decision splits. Owning the hardware does not make on-premise cheap.
Whether it is 5 billion won of hardware or 300 dollars an hour of rental, the cost runs whether you use it or not. An API, by contrast, charges only for the tokens you spend. So for on-premise to work arithmetically, workload has to arrive without pause and keep the GPUs running as hard as they can. For a company that uses it in bursts, the math does not close.
This is why you must not invert the order of the decision. Start from 'our data is sensitive, so let us go on-premise' and you end up with hardware you cannot fill. Start instead by asking whether your traffic can sustain high utilization, and most companies arrive quickly at the conclusion that full on-premise is not the answer.
That does not mean on-premise is unnecessary. It means the answer is partial, not total.
Open weights reaching frontier class is the precondition
The reason this conversation did not hold until recently is simple. Open-weight models were not in the same bracket as the top models. They compared to the previous generation.
That line was crossed recently. Kimi K3 arrived with frontier-class scores and is being used heavily on OpenRouter, with an open-weight release announced. Qwen has officially previewed the 2.4T-class Qwen3.8 Max Preview, and GLM and DeepSeek are expected to follow. Which is to say the proposition of running top-class performance in house has started to hold.
Price moves with it. Once weights are public, many GPU rental providers host the same model and compete on inference token price, so the price keeps falling. GLM 5.2 already got cheaper that way.
But looking only at per-token price misleads. A model with a cheap unit price that burns far more tokens to finish the same job is more expensive in the end. So compare cost per task, not cost per token. The same applies to an on-premise evaluation: a quote only means something once you know how many tokens your in-house model needs to finish the same work.
So what do you do first
Take this order.
First, list the data and the work that must not leave. The criterion is path exposure, not training. Include the analysis you would have to be able to run on the day of an incident.
Second, run only what is on that list in house, on open weights. You do not need to stand up a full frontier-class model. If the use is narrow — incident response, sensitive document handling — you can start with something far smaller, and that is exactly the shape of what Hugging Face did on the day.
Third, leave everything else on frontier APIs. Competition has pushed prices and promotions in the user's favour, so moving work inside that has no reason to be there is a loss.
Fourth, judge full on-premise on utilization alone. Measure whether your workload actually arrives fast enough to keep GPUs at maximum, and only then get a quote. Reverse that order and you are left with idle hardware.
The bottleneck further down is power
To look at on-premise cost over a long horizon you have to go one level down. GPU price is an output of the supply chain beneath it.
We are in a memory-bound stretch, so component prices like HBM are sharply up. Relieving that requires production capacity, which is bound to the build time of data centres and fabs, and to the annual output of ASML's lithography machines. That the future allocation of those machines is effectively already settled is what explains today's token prices.
Bottlenecks do get routed around, though. Inference-specific chips skip the newest process and raise efficiency by restricting what they do, and you can always cluster a larger number of previous-generation chips. Capital buys a way around. Keep going down and the last bottleneck standing is energy — power.
So over a long horizon it is right to expect token prices to keep falling like an electricity bill. A company pricing out on-premise today will get different numbers pricing it out three years from now. One more reason not to rush a full migration.