Small models, big estates: the case for encoder-only transformers in production

The public conversation about AI is a conversation about frontier models: the largest generative systems, benchmarked on reasoning, priced per token. The daily reality of enterprise AI is different. Most of what large organizations actually need from language models is understanding at volume — classify this ticket, route this complaint, flag this transaction narrative, extract the sentiment of this review, score this document against this policy. These are representation problems, and for representation problems the encoder-only transformer remains a remarkably stubborn optimum.

Our center’s recent work on fine-grained, document-level sentiment mining over big data — built on ModernBERT, the current state of the art in encoder architectures — sits squarely in this territory, and the lessons generalize well beyond sentiment.

Why encoders keep winning their lane

Three properties make modern encoders the right tool for high-volume understanding tasks.

Economics: an encoder that runs on a single GPU — or CPU fleet — processes millions of documents for the cost of a frontier model’s rounding error. When the workload is continuous rather than occasional, per-inference cost is the entire business case.

Latency and placement: encoders are small enough to run inside your own perimeter, next to your data. For regulated estates — banks, telcos, health systems — this dissolves an entire category of data-residency and vendor-risk questions rather than answering them.

Determinism and evaluation: classification with a fixed label space can be evaluated the boring, rigorous way — held-out sets, confusion matrices, drift monitoring. You can tell a model-risk committee precisely how often it is wrong and in which direction. Try that with a generative system’s free-text output.

The post-training gap

The catch is that an encoder is not useful off the shelf; it becomes useful when post-trained on your task and your data. This is where most enterprise attempts stall — not on modeling technique but on the unglamorous substrate: labeled data of adequate quality, GPU capacity for post-training runs over genuinely large corpora, and honest held-out evaluation. In our own experiments, access to dedicated, data-center-grade GPU capacity was the difference between exploiting a big-data corpus and merely sampling it. That constraint shapes architecture choices more than any benchmark does.

Not either/or, but a portfolio

None of this is an argument against frontier models, which we also deploy where generation, synthesis, or open-ended reasoning is genuinely required. It is an argument for treating model selection as an engineering decision rather than a fashion decision. The mature estate looks like a portfolio: small, post-trained encoders handling the high-volume understanding workloads; larger generative models reserved for the tasks that need them; and routing logic in between.

What to do about it

Audit your AI roadmap by workload shape. Every use case that reduces to classify, extract, score, or route is a candidate for an encoder at a fraction of the projected cost — with better auditability thrown in. Prototype one such workload end to end, including post-training and evaluation, before renewing the assumption that every problem needs the largest model on the market. The savings tend to fund the rest of the roadmap.