Small models, big leverage: what MiniLM-class models do on one VM

The problem

Ask most teams what "adding AI" means today and you'll hear the same short list: a frontier model, an API key, and a per-token bill. The industry conversation has collapsed onto one axis — how big is the model, and how much does each call cost — as though intelligence were the only ingredient that mattered and bigger were always the answer.

It's a mistake, and it's an expensive one. A large share of the AI work a business actually needs isn't the kind a frontier model is for. "Which past cases look like this one?" "Are these two customer records the same person?" "Is this ticket about billing or shipping?" "Which paragraph of our own documents answers this question?" None of these require a model that can write an essay or reason through a proof. They require a model that can turn text into a number that means something — and that is a job a small model, running on one modest VM, does better, cheaper, and more privately than any API call ever will.

You do not need a frontier model to embed documents or match records. It's worth being precise about what the small models are for, where they earn their keep, and where they genuinely fall short.

Why it matters now

The unlock is that small, high-quality embedding models — the MiniLM-class family of sentence encoders and their relatives — got good enough to be boring. They map a chunk of text to a vector: a list of numbers positioned so that text with similar meaning lands close together. That single operation is the engine underneath a surprising amount of "AI" you actually want:

• Semantic search over your own records — find the right chunk by meaning, not keyword match.
• Similarity and matching — "find the case like this one," deduplication, "is this the same vendor under a different spelling."
• Clustering and routing — group tickets, tag inbound, send a request to the right queue.
• Classification and risk scoring — often with an even smaller model sitting on top of the vectors.

What these have in common is that the model is asked to measure, not to reason or write. That distinction is the whole point. A small embedding model doesn't understand your business, and it can't compose a paragraph. It places text in a space where distance means similarity. Reasoning and writing are the large language model's job — a genuinely different and genuinely more expensive one. Conflating the two is how teams end up paying frontier prices to answer a question a CPU could have answered locally.

It helps to notice how much of day-to-day operational software is really the measuring kind. A support desk routing tickets, a CRM flagging probable duplicate contacts, a records system surfacing the three prior matters that resemble the one on your screen, a pipeline deciding whether an inbound document is an invoice or a statement — none of these are asking a model to think. They're asking it to compare, and comparison is cheap. The frontier model is the right tool for the rarer moments when something genuinely has to be composed or reasoned through; the trouble is that the industry reaches for it as the default tool, and then bolts a per-call bill onto work that never needed it.

And "small" here is not a compromise you tolerate — it's the feature. A MiniLM-class encoder fits comfortably on commodity CPU. It embeds locally, so the part of your pipeline that touches your entire corpus never leaves the machine. There's no per-token meter running while it works. Scale it up and the cost curve stays flat, because you're spending fixed local compute, not a linear cloud bill that grows with every record you index. The properties that make it "not a frontier model" are exactly the properties that make it the right tool for this class of work.

How FastYoke approaches it

FastYoke runs this pattern in two shipped features, and neither one reaches for a frontier model to do the small-model job.

Yoker embeds your records on your own VM. When you ask Yoker, FastYoke's built-in assistant, a question about your data, it doesn't send your corpus anywhere to search it. Each record chunk is embedded with an ONNX model on your FastYoke VM, and the vector search runs in Rust against your own tenant-scoped chunks — no external embedding API, no tenant data handed to a provider just to find the relevant rows. That's the retrieval-then-optional-synthesis pattern in practice: the small model finds the right chunks locally, and only if you've opted into synthesis does a language model ever see text — and then only the matched, relevant, PII-scrubbed slice, never the whole corpus. Every AI write stays gated behind your approval. The mechanics of the full loop — enabling it, seeding the corpus, and reading the citations that prove an answer came from your data — are covered in the RAG assistant recipe.

No-Show ML shows that classical models still count. Not every useful model is a neural net. Yoke's no-show predictor is plain logistic regression, trained on your own completed and no-show jobs and scored on your VM. Its features are unglamorous and effective — prior no-show count, booking lead time, whether it's a first visit — and it trains on a modest history of past jobs, retraining on a weekly cadence. It is not deep learning, it does not need a GPU, and it is exactly the right tool for the question it answers: which customers on this week's bookings are likely to no-show. A tiny classical model, run locally on your own data, earns its keep every week.

The thread connecting both: the small model does the measuring on your hardware, on your data. A large external model, if it's involved at all, is optional, scrubbed, and downstream of everything sensitive — the AI posture that governs the whole platform.

What to watch out for

None of this is a case for using small models everywhere out of principle. Take the argument seriously enough to know its limits.

Small models have real quality ceilings. A MiniLM-class embedding captures general semantic similarity well, but it won't grasp the deep domain nuance a larger or fine-tuned model would — the distinction between two contract clauses that read almost identically but mean opposite things, say. If your matching problem hinges on exactly that kind of nuance, a small general-purpose encoder will quietly get it wrong, and "it runs locally" is no consolation.

Embedding quality is downstream of your data and your chunking. The model is only half the system. How you split records into chunks, how much context each chunk carries, whether you're embedding clean text or formatting noise — all of it shapes what "similar" ends up meaning. The same model can produce excellent retrieval on well-chunked data and useless retrieval on badly-chunked data. Most disappointing results trace back here, not to the model.

Evaluation is not optional. Garbage similarity is worse than no similarity, because it looks like it's working. A retrieval system that confidently returns the wrong chunks will mislead every downstream step that trusts it. You need a small evaluation set of real questions with known-good answers, and you need to actually measure hit rate against it before and after any change — not vibe-check a few queries and ship.

"On-VM and small" doesn't excuse skipping measurement. The privacy and cost wins are real, but they're orthogonal to whether the thing is any good. Local compute buys you sovereignty and a flat bill; it does not buy you correctness. That still has to be earned the ordinary way, with evaluation and iteration.

Where this goes next

The useful mental model is a division of labor, not a hierarchy. Small models measure — they turn your text into vectors and your history into predictions, on your own hardware, at fixed cost, with your data never leaving the box. Large models reason and write, when and only when the work genuinely calls for it, over a narrow and scrubbed slice of context. Most of the AI value in an operational business lives on the first side of that line, which is precisely the side the industry's frontier-model conversation keeps overlooking.

The economic shape follows from the technical one: a small on-VM model turns a recurring, volume-scaled cloud bill into fixed local compute you already own. For the retrieval, matching, routing, and scoring that make up the bulk of real business AI, that's not a downgrade — it's the better engineering answer.

If you want to see the pattern end to end, Yoker covers the on-VM retrieval model, the broader AI posture explains gated writes and local-first retrieval as policy, and on-prem covers what running all of it inside your own network looks like.