On-VM AI, no cloud round-trip: face recognition and RAG that never leave your VM

The round-trip you didn't mean to add

You have a workflow step that needs a model. A record gets classified before it moves to the next state; a batch job scores a few thousand rows overnight; a question needs answering from data that lives in one tenant's database. The obvious build is a call to a hosted inference API — it's a POST away and the model is excellent.

The moment you do that, you've added two things to a step that used to be local. First, a network hop on the critical path: the transition can't fire until a remote endpoint answers, so you've inherited that endpoint's availability, rate limits, and tail latency as failure modes you now design around. Second, a data-egress event: whatever context the model needed just left your network, and if the data was sensitive, that's a fact you now have to characterize for an auditor.

For an AI feature sitting in front of a person — a chat box someone types into — neither cost matters much. A few hundred milliseconds is invisible next to the seconds a human spends reading, and the person chose to ask. For AI sitting inside a workflow, both costs are real and both are avoidable by moving the model to where the data already is.

One clarification up front, because the title invites it. FastYoke does not ship a face-recognition product you can click; treat detection and recognition here as the general class of on-VM inference the same architecture supports — a model a developer loads and runs where the data lives, exactly as the retrieval assistant runs its embedding model — not a feature on a pricing page. The shipped, nameable examples below are retrieval (Yoker) and a small operational model (No-Show ML). The image case is the pattern extended, not a claim about what you can buy.

Why running it locally got practical

None of this is new as an idea; what changed is that acting on it stopped requiring a research team.

Small models got genuinely good at the operational jobs — extraction, classification, ranking, answering a grounded question over a handful of retrieved records. These are not frontier problems and never were. Embedding, in particular, is a small, fast operation that runs fine on commodity CPUs, so the part of any RAG pipeline that touches your whole corpus can stay in-house at almost no cost.

Just as important, the runtime got boring in the good way. A model exported to ONNX runs in-process through a pure-Rust inference engine on CPU — no GPU, no accelerator, no sidecar service, no exotic OS. On FastYoke that engine is tract, the same one whether the binary runs in the cloud, on-prem behind a firewall, or on an edge node in a truck cab. "Run the model on the VM" is no longer a euphemism for "stand up a GPU cluster next to the app."

What on-VM inference actually requires

Three things, worth stating plainly because the honest version of this argument lives in the constraints.

A model that fits the hardware. CPU-friendly, modest memory, quantized where it helps. This rules out hosting a frontier LLM yourself — and that's fine, because the operational jobs above don't need one. Pick the smallest model that clears your quality bar.

An in-process runtime. Something that loads the model and runs inference inside the same process as your application, with no network call. ONNX plus a CPU inference library is the well-trodden path; the point is that the inference is a function call, not an RPC.

Tenant-scoped data it can read locally. The model reads from the same store the workflow reads from, under the same isolation rules — so on a multi-tenant platform every read the inference path makes is scoped to one tenant, and the model can't see across the boundary because the query feeding it can't.

Get those three and the latency win is a property of the architecture, not a tuning exercise: there is no hop to be slow.

Where the latency win is real (and where it isn't)

Be precise about this, because it's the claim people oversell.

The win is largest when the model sits inside a state transition or a tight batch loop — a guard that fires thousands of times a day, a per-record classifier, retrieval over an operational corpus. Every one of those calls would otherwise be a remote round-trip with its own tail, and removing the hop removes a whole category of "what happens when the API is slow / down / rate-limited at 4pm" design work.

The win is smallest for the human-facing chat box — exactly the surface people demo first. If your only AI feature is a sidebar someone types into, latency is not your argument; sovereignty might be. Don't build the harder local path to win milliseconds a human will never perceive.

The privacy win is a property, not a promise

The other half is data egress, and the distinction here is the whole point. "We don't send your data anywhere" from a vendor is a promise — a commitment that can be revised at renewal or compelled by a jurisdiction. "The data never left the VM" is a property of the network: nothing crossed the boundary because the code path has none to cross. The honest test is whether you can cut the cable and have the inference keep working. If yes, you hold the property; if the model is a remote call, you hold the promise, however good the contract behind it.

There's a pragmatic middle that captures most of the benefit without self-hosting a large model: split retrieval from synthesis. Run retrieval locally over your own corpus — the part that touches everything — and, if you need a strong generative answer, send the narrow slice of matched context out for synthesis, scrubbed. Your corpus stays put, and the thin slice that leaves is one you can characterize exactly.

How FastYoke runs it

Three shipped surfaces make the pattern concrete.

Yoker (on-VM RAG). Yoker is the docked assistant that answers questions about a tenant's own entity records and text attachments. It embeds record chunks with an ONNX model on your VM and runs vector search in Rust against tenant-scoped chunks — there is no external embedding API, so the corpus is never shipped out for search. Answers cite the exact source records that grounded them, and if retrieval finds nothing relevant it declines rather than inventing. External LLM synthesis is optional and PII-scrubbed; retrieval stays local regardless, and every AI write is gated behind your approval. Yoker is à la carte on Pay-as-you-go and included with Enterprise Platform.

No-Show ML. This is the clearest example of on-VM inference as opposed to retrieval. It's a logistic-regression model trained on your own completed and no-show jobs and scored on your VM — features as simple as prior no-show count, booking lead time, and a first-visit flag. It trains on a couple hundred historical jobs, retrains weekly, and surfaces as a brief listing the at-risk customers on upcoming bookings. No GPU, no external scoring service, no customer data leaving to a model host: the training set and the model both live where the jobs do.

Local guard evaluation. The workflow engine evaluates its guards locally, so a state transition makes no cloud API call to decide whether it may fire — the same principle one layer down, and why the whole platform can run air-gapped with no outbound traffic at all.

The image case rides on the same three requirements. A developer who needs detection or recognition — matching an incoming document against known templates, flagging anomalous records, any benign classification over local inputs — loads an ONNX model and runs it through the same in-process engine that powers Yoker's embeddings, reading tenant-scoped data locally. That path isn't special-cased; it's the extensible shape, not a product.

What to watch out for

Take the argument seriously enough to know its costs.

Model lifecycle is the hard part. A model you can't reach is a model you can't update. In an air-gapped estate new weights arrive as an operator-pulled artifact on a schedule someone owns, with a rollback someone has tested. That's a process commitment, not a docker pull.

Evaluation without telemetry is harder. You can't watch aggregate quality across machines you can't see. You need local evaluation sets, local scoring, and a human review loop, built up front — retrofitting evaluation onto a running system is miserable.

Hardware sizing goes wrong in both directions. Undersize and the feature is slow enough that people route around it; oversize and you've bought idle capacity. Size to sustained load, not peak-hour.

Don't ship a weak local model out of principle. If a strong hosted model would serve users materially better and the data classification permits it, use it — the retrieval-local, synthesis-remote split exists precisely so you don't have to choose all-or-nothing. Sovereignty is a constraint to satisfy where it applies, not a score to maximize everywhere.

Where this goes next

The destination isn't "all inference moves on-device." It's a split that follows data classification rather than fashion: the parts that touch the whole corpus and sit inside tight loops — retrieval, per-record scoring, guard evaluation — run local, because that's where the sensitive data and the latency budget are; heavy generative synthesis is purchased where it's worth it and the classification allows it. The interesting engineering question stops being "cloud or local" and becomes which slice of context is allowed to cross the boundary — a question with a precise, auditable answer. If you're weighing it, the useful first move is architectural: find the inference calls that sit inside a transition or a batch loop, and move those before you touch the human-facing chat.

Yoker covers the on-VM retrieval model end to end, On-Prem covers running the whole platform with no outbound at all, Edge covers the same engine on hardware in the field, and the broader AI posture — gated writes, local retrieval, optional and scrubbed external models — is the policy all of it follows from.