On-prem AI is back: sovereignty, latency, and cost in the LLM era

The problem

A hospital system wants an assistant that can answer questions about its own patient records. A regional bank wants the same thing over its loan files. Both projects reach the same meeting, where someone from legal asks a question that sounds simple and isn't: where does the data go?

The vendor's answer is usually a good one. Encryption in transit and at rest. A signed agreement. A clause stating your data will not be used to train the model. Regional processing. Those are real controls backed by real contracts, and for most organizations they settle it.

For a subset of buyers the answer lands wrong, and it takes a while to say why. It isn't that they doubt the vendor. It's that they're being handed a promise when their obligation requires a property. "We don't train on your data" is a commitment someone made and can, in principle, break, revise at renewal, or be compelled to break by a jurisdiction they operate in. "The data never left the building" is a fact about the network. Those are different objects, and no amount of contractual strengthening turns the first into the second.

That gap is why on-prem AI — an idea most people filed under "legacy" a decade ago — is back on serious architecture agendas. Not nostalgia, and not paranoia. Three forces are pushing it, and they push with very different strength depending on what you're building.

Force one: sovereignty is a property, not a promise

Start with the strongest of the three — and the one most often mischaracterized as fear. Data residency has been a contractual object for a decade: you sign a document, the vendor commits to processing in a region, auditors accept the document as evidence. What's changed is that the regulated side of the market increasingly needs the physical version of the same claim. GDPR and the EU/UK localisation regimes, India's DPDP, China's PIPL, HIPAA obligations around protected health information, the OT/IT boundary rules utilities operate under: in each case a growing share of buyers find the cleanest way to satisfy the requirement is to remove the question rather than answer it well.

AI made this sharper. A conventional integration sends structured records to a known endpoint on a known schedule; a language model in the loop sends whatever the prompt needed — free-text context assembled at runtime, the exact shape of data that's hardest to draw on a data-flow diagram. Auditors have noticed.

One clarification, because it gets blurred constantly: air-gapped does not mean "reduced egress" or "telemetry off by default." It means literally no outbound. The honest test is whether you can cut the network cable after deployment and have everything keep working — workflows, audit log, retrieval, the lot. Anything less is a lower-egress cloud deployment, which may be perfectly adequate but shouldn't be sold as the other thing.

Force two: latency is a workflow problem, not a chat problem

Sovereignty explains why buyers want on-prem. Latency explains where it actually changes the product.

For a chat box in front of a human, a cloud round trip is a non-issue — several hundred milliseconds is invisible next to the seconds a person spends reading the answer. If your AI feature is a sidebar someone types into, latency is not your argument.

The picture inverts when the model sits inside a workflow rather than in front of a person. A state transition that must consult a remote endpoint before it can fire has inherited that endpoint's availability, rate limits, and tail latency as its own — and every remote call becomes a failure mode you design around. What happens to the transition when the API is slow? Down? Rate-limited at the worst possible hour?

Local evaluation removes the hop from the critical path, and with it a whole category of design work. The largest wins are in tight loops — per-record classification, retrieval over an operational corpus, guard conditions firing thousands of times a day. The smallest wins are in exactly the place people demo first: a human-facing chat box.

Force three: cost has a crossover, and it's a real calculation

Per-token pricing is genuinely excellent at low and spiky volume: you pay nothing when idle, scale instantly, and never buy a GPU that sits dark for eleven months. For most workloads that's simply the right answer, and the on-prem case shouldn't be argued as though it weren't.

The economics change under steady, high, predictable volume. Metered pricing scales linearly forever while owned hardware is a fixed cost you amortize, so there's a crossover point where fixed wins — and the important thing about that point is that it's a calculation with your own numbers in it, not a slogan. It needs honest inputs: sustained tokens per day rather than peak-hour extrapolations; hardware over a realistic replacement cycle; power, cooling, and rack space; and the engineering time to operate the thing, which is the line people leave out and which is rarely small.

Be equally honest about the failure case. On-prem trades a variable bill for fixed capital plus an operating burden that doesn't go away, and idle GPU capacity is pure waste — you paid for it whether or not you used it. If your volume is spiky or hasn't stabilized, metered pricing is still your friend. Cost is the weakest of the three forces for most organizations, and the one most often used to justify a decision that was really about sovereignty.

What actually got practical

None of the three forces is new. What's new is that acting on them stopped requiring a research team.

Small models got genuinely good. Open-weight models you can run on modest hardware handle what operational software actually needs — classification, extraction, summarization, answering a grounded question about a handful of retrieved records. They are not frontier models and you shouldn't pretend otherwise. But "is this ticket a billing issue or a shipping issue" was never a frontier problem.

Embeddings are cheap to run locally. This is the quiet unlock. Embedding is a small, fast, well-understood operation that runs fine on commodity CPUs — so keeping it local moves the part of the pipeline that touches your whole corpus in-house at very little cost.

The hybrid split captures most of the benefit. You don't have to choose between hosting a frontier model yourself and sending everything to a provider. Run retrieval locally over your own corpus; optionally send only the matched context out for synthesis, scrubbed. Your corpus stays put, and the narrow slice that leaves is one you can characterize precisely — exactly what the auditor was asking for.

How FastYoke approaches it

FastYoke's on-prem deployment is the same single Rust binary that runs the cloud platform, installed inside your network — not a reduced edition. The workflow engine evaluates its guards locally, so a state transition makes no cloud API call, and the audit log, the compiled workflow rules, and the sandboxed scripting tier are all self-contained. Tenant databases are per-tenant files on your own hardware — each client's data a discrete file under your control rather than rows commingled in a shared database an auditor has to scope. Air-gap means what it says: the binary does not phone home, and you can cut the cable.

The AI layer follows the hybrid split. Yoker, the built-in assistant, embeds record chunks with an ONNX model on your VM and runs vector search in Rust against tenant-scoped chunks — no external embedding API, and tenant data is not sent to a provider for search. Answers cite their sources. External language models remain optional and PII-scrubbed by default; retrieval stays local regardless. And every AI write is gated — nothing changes until you approve it. Yoker is à la carte on Pay-as-you-go at $299/mo metered, and included with Enterprise Platform.

What to watch out for

Take the argument seriously enough to know where it's wrong.

Model lifecycle is the hard part. A model you can't reach is a model you can't update. In an air-gapped estate, new weights arrive as an operator-pulled artifact on a schedule someone owns, with a rollback path someone has tested. That's a process commitment, not a download.

Evaluation without telemetry is harder. You can't watch aggregate quality across a fleet you can't see. You need local evaluation sets, local scoring, and a human review loop — built up front, because retrofitting evaluation onto a production system is miserable.

Hardware sizing goes wrong in both directions. Undersize and you ship something slow enough that people route around it. Oversize and you've bought idle capacity, which is the exact failure the cost argument was supposed to avoid.

Don't run a weak local model out of principle. If a strong hosted model would serve the user materially better and the data classification permits it, use it. Sovereignty is a constraint to satisfy where it applies — not a score to maximize everywhere.

On-prem is an operating commitment, not a purchase. Upgrades, backups, monitoring, incident response, certificate renewal: you own all of it. Organizations that fail at on-prem usually fail here, not at the install.

Where this goes next

The destination isn't "everything moves back on-prem." It's a split that follows data classification rather than fashion: retrieval and the operational loop stay local, because that's where the sensitive corpus and the tight latency budget are; frontier synthesis is purchased where it's worth it and the classification allows it. Most organizations will run both, and the interesting engineering question becomes which slice of context is allowed to cross the boundary — a question that has a precise answer, which is the whole point.

If you need the property rather than the promise, the useful next step is arithmetic, not philosophy: write down your sustained volume, your data classification, and the ops headcount you can actually commit — then see which of the three forces is doing the real work for you.

The concrete version: On-Prem covers the deployment shapes and the honest tradeoffs, Yoker covers the on-VM retrieval model end to end, and the broader AI posture — gated writes, local retrieval, optional and scrubbed external models — is the policy all of it follows from.