Building a Self-Sovereign AI Training Loop: Improving Models Without Distilling From Frontier Providers

By Aaron Gammon · Founder, Inferrex · June 2026

This is the most technical thing I've written for this collection, and it comes with the strictest version of my standing rule: I'll share principles and outcomes, never the methodology. The specific recipes — how datasets get built, how layers are structured, where the thresholds sit — are the proprietary core, and I sell what the factory produces, not the schematics. So you'll find arguments and results here, and deliberately no blueprint.

I'm writing it because the position itself is uncommon enough to be worth defending in public, and because I'd want to read it if someone else had taken this path. Live platform figures, including model accuracy, are at inferrex.com/claims.

There's a well-worn path for making a small, specialised model good: you have a big frontier model generate the training data, and you distill its outputs down into your smaller one. It works, it's fast, and it's what most teams reach for without much deliberation. The big model becomes the teacher; your small model becomes a compressed echo of it.

I understand the appeal. I also think, for what I'm building, it's a trap — and the costs are the kind that don't show up until they're expensive to undo.

The first cost is dependency. If your model is only as good as the frontier model that taught it, you've quietly made a frontier provider a permanent input to your product. Your improvement loop runs through their API, on their terms, at their prices, under their licence. You don't own your own progress; you rent it.

The second is a hard ceiling. A distilled model inherits its teacher's blind spots and can't easily exceed it on the things you actually care about, because it was trained to imitate rather than to be correct on your problem. You've capped your own quality at "slightly worse than the teacher," forever.

The third, and for me decisive, is sovereignty. You cannot air-gap a model whose quality depends on calling out to a frontier provider. The moment "getting better" requires the internet and a third party, you've forfeited the ability to run with zero external dependency — which, for the buyers I care most about, is the whole game. I've argued elsewhere that you can't air-gap a dependency you don't own; this is that principle applied to the model layer itself.

So we built the other thing: a closed improvement loop with no frontier model anywhere in it.

The model improves from two sources only. The first is our own corpus — a large, growing body of real API specifications, the actual structures the model exists to understand. Not a frontier model's opinions about those structures, the structures themselves. The second is human corrections — when the system gets something wrong and a person fixes it, that correction feeds back as ground truth. That's the entire diet: the real data, and humans being right about it. No teacher model, no distillation, no synthetic data laundered from a frontier provider.

This is harder. It's much easier to ask GPT-or-similar to label ten thousand examples than to build a corpus and a correction loop that earn their ground truth honestly. But the result is a model that improves on its own terms, owns its own ceiling, and — crucially — keeps improving in an environment with no internet at all, because nothing in the loop reaches outside the box. "Self-sovereign" isn't a branding flourish here. It's a literal architectural property: the loop is closed, so it can run inside the wall.

Here's a principle that cost us something to learn, offered as a lesson rather than a recipe.

The intuitive way to decide which of a model's outputs to trust is to look at its confidence — keep the high-confidence answers, flag the low-confidence ones for a human. It's the obvious approach, and on familiar data it's even roughly right.

It breaks exactly where it matters most: on the genuinely novel field, the one unlike anything the model has seen before. There, token-level confidence stops tracking correctness — a model can be serenely, fluently confident about something it has fundamentally misread, because confidence measures fluency, not accuracy, and the two come apart precisely on the out-of-distribution cases you most need to catch. Trusting confidence on novel inputs means trusting the model most exactly when it's most likely to be wrong.

The mitigation we use — and I'll state the principle, not the mechanism — is to check a new judgement against what's already known to be true rather than against the model's feeling about itself. Disagreement with established ground truth is a far more reliable alarm than low self-reported confidence, because it's anchored to reality instead of to the model's mood. If you take one transferable idea from this piece, let it be that: on anything novel, trust agreement-with-known-truth over the model's confidence in itself.

Now a result, stated as a result.

The reflex in the industry is that bigger is better — that a larger, general-purpose model will outperform a smaller one on basically anything. For a narrow, well-defined domain, that reflex is wrong, and we have it the right way round on purpose. A smaller model, specialised hard on a specific domain and trained on the right data for it, outperforms a much larger general-purpose model attempting the same task cold — at a fraction of the cost to run.

The intuition is straightforward once you say it out loud. A general model has to be mediocre at everything to be passable at anything; it spreads its capacity across the whole world. A specialised model spends all of its capacity on your problem. For understanding the structure of business APIs — a deep but bounded domain — depth beats breadth decisively. This is also what makes the sovereignty story economically real: a right-sized specialised model is small enough to actually run on a customer's own hardware, which a frontier-scale general model is not. The specialisation and the sovereignty reinforce each other — the same choice that makes it better also makes it deployable where it has to be.

Pull the threads together and the architecture has a single throughline: every choice is made so the loop can run with no external dependency.

No frontier distillation, so improvement doesn't require an external provider. A corpus-and-corrections diet, so the ground truth is owned and local. Specialised-and-small, so the model is light enough to live on the customer's hardware. Agreement-with-known-truth over self-confidence, so quality control doesn't need to phone anything either. None of these is an isolated cleverness. They're the same decision made repeatedly — keep the loop closed — because a model that can only get better by reaching outside the box can never run inside one. And running inside the box, for sovereign and air-gapped deployments, isn't a feature. It's the requirement everything else has to serve.

I'll be honest that this path is more work, and that there are quarters where the distillation shortcut would have been tempting. But the shortcut builds a dependency you can't remove later, and a ceiling you can't raise, into the deepest layer of the product — and it forecloses the one thing the most serious customers actually need, which is for the whole thing to run, and keep getting better, inside a box with no internet.

So we don't distill from frontier models. The models improve from real data and human truth, on their own terms, owning their own ceiling, phoning nothing home. That discipline is invisible in a demo and decisive in a sovereign deployment — which is exactly the kind of trade-off the whole company is built to make.