The free model at the front of the line
A decision model small enough to run on a Mac mini, with open weights and no bill, can in principle take the first look at every request and pass only its doubts to a paid model. This entry is about when that cascade pays: what the free tier has to get right, how to tell whether its confidence can be trusted, and what one two-hour fine-tuning run on free GPUs changes. The evidence is three experiments on my own hardware against the paid model from Build Log No. 005 and a frontier model, for $5.45 in total API spend.
A cascade answers the easy requests cheaply and saves the expensive model for the hard ones.
Most requests a piece of software has to judge are easy. A few are not. A cascade puts the cheapest capable model first, lets it answer when it is sure, and passes everything else up a tier. The saving is every request the cheap tier keeps; the risk is every one it keeps and gets wrong.
Build Log No. 005 made the case for a decision model as a gate in front of an expensive one. The paid decision model there, Jev, costs about two cents per thousand answers. This entry asks the next question down: can a tier that costs nothing at all sit in front of that?
LAYA is an open-weights look-alike of the paid decision model, and it runs on a desk.
It takes the same request: a piece of text plus typed questions — yes/no, pick one, or rate it — and returns an answer with a probability per option, in the same wire format. It never writes prose, so there is nothing to parse and nothing to hallucinate.
It is a 421-million-parameter encoder (ModernBERT-large) with a decision head, released under the Apache 2.0 license.[1] It ships as three checkpoints — English, multilingual, and one fine-tuned on a published typed-decisions benchmark — plus a Route Mode that picks between the first two by detecting the language of each request. Every experiment here started from that recommended default. On an M4 Mac mini's GPU a single question comes back in about 50 milliseconds, and the only cost is the electricity.
Accuracy, honest confidence, and the price gap to the next tier.
It has to be right often enough to be worth asking. A first tier that is wrong on half its answers mostly adds latency before the real answer.
Its confidence has to be honest. The cascade trusts the free tier only when it says it is sure. If "90% sure" is right 64% of the time, the threshold lets through confident mistakes that nothing downstream ever sees.
The tier behind it has to be expensive enough to save. Keeping three-quarters of the calls from a model that costs two cents per thousand saves a cent and a half. Keeping them from one that costs twelve dollars per thousand is a different decision.
The next three tabs take those in order: the free model as shipped, the free model as a gate, and the free model after one fine-tuning run.
The thread
The first question is the plainest one: taken straight off the shelf, how often is the free model right, and how does that compare with the two paid ones?
As shipped, the free model was right about half the time.
The test set is 300 questions across six kinds of decision, 50 of each, every answer key a human label from a public dataset. All three models got the same question.
The frontier model (Claude Fable 5.1, at high effort) was the most accurate and a hundred times slower than the local model. The paid decision model sat between them on accuracy, at a price that rounds to nothing. LAYA's fine-tuned checkpoint did only slightly better than the default, because it was trained on a different set of workflows.
It holds up on routing-shaped questions and falls apart where a judgment needs a fact or a chain of reasoning.
Its best results were the two questions that look most like triage: which of twenty support intents a customer message expresses, and whether a message is a prompt-injection attempt. The widest gaps to the paid models were on the ones a person answers by knowing something or thinking a step further — whether a premise settles a claim, and multi-hop yes/no questions. Rating a review on five levels, it landed near chance.
A 52% that is not a score
On the multi-hop set the free model answered "no" to 49 of 50 questions. The set is half "yes," so 52% is the base rate, not reasoning. Its own documentation warns that its yes/no head can follow its option labels instead of the text; asking the same question as a two-option choice with neutral labels moved it only to 43 "no" answers of 50.
As shipped, its "90% sure" was right 64% of the time. One offline fix closes the gap.
Calibration is whether stated confidence matches the hit rate. For a cascade it matters more than accuracy, because confidence is the only thing the router reads.
The fix is called temperature scaling: stretch or squeeze the model's probabilities by one number per question type, fitted on held-out data. It needs no retraining and no new calls — the stored probabilities are enough. It does not make the model more accurate. It makes the model honest about how accurate it is, which is what a threshold needs.
Two numbers that are easy to mix up
The package's own confidence field is not "chance I'm right." It is one minus the
normalized entropy of the probabilities, a measure of how spread out they are. Every analysis
here stores both and routes on the top probability.
The thread
A first tier does not have to be accurate on everything. It only has to be accurate on the requests it keeps. This tab measures what keeping them costs.
Held to within one point of the paid model, the stock free model could keep about one request in eight.
Instead of choosing one threshold, the sweep tried every level from "keep nothing" to "keep everything" and recorded what each one did to final accuracy and spend.
The curve has no plateau. From the first handful of requests onward, each extra slice the free model keeps costs accuracy at a roughly steady rate. Keeping half the calls would have saved 59% of the frontier bill and cost 11 points. Adding other signals — the margin between its top two answers, whether two wordings of the same question agreed, the question type, or a learned combination of all of them — did not beat the recalibrated top probability on 300 questions.
The paid decision model in front of the frontier model matched frontier accuracy at 45% less cost. The free model added nothing to it.
| Routing policy | Accuracy | Cost per 1,000 | Mean time |
|---|---|---|---|
| Frontier model for everything | 79.7% | $11.85 | 5.7 s |
| Jev first; frontier only when Jev is under 95% sure | 79.7% | $6.49 | 3.6 s |
| Jev first; frontier only when Jev is under 90% sure | 78.3% | $5.47 | 3.1 s |
| Free model → Jev → frontier (best cross-validated) | 77.7–78.7% | $5.47–10.47 | 3.2–5.7 s |
| Jev for everything | 71.3% | $0.019 | 0.26 s |
With the threshold chosen on four-fifths of the questions and scored on the remaining fifth, the best three-tier setup put 0% of traffic on the free model. The paid decision model was already doing the free tier's job, for a price the free tier cannot meaningfully undercut.
The thread
The free model as shipped is a starting point, and its author says so. This tab trains it on one real job and runs the cascade again.
The model's own documentation calls the stock checkpoints a starting point, not a finished product.
In the author's words, "a fast base to specialise, not a zero-shot decision engine."[1] Their fine-tuned checkpoint went from roughly 36% to 77% on its own benchmark.
The job chosen for training was one the paid decision model had already been tuned for in Build Log No. 005: reading a GitHub repository's name, description, language and topics, and deciding whether it is about AI. The paid model answers it as seven small yes/no questions combined in code, and it scores 98.7% against blind frontier-model labels. That made it a clean test. The free model could learn from the paid model's own answers, a technique called distillation, and be scored on repositories neither had been tuned on.
The paid model wrote the lessons for 54 cents. Kaggle's free GPUs did the training.
The whole run used LAYA's own published training script, with only the data loading swapped.
| Run | Repos | Examples | Epochs | GPU training time | Whole job | Exam accuracy |
|---|---|---|---|---|---|---|
| Pilot | 300 | 2,100 | 1 | 3 m 3 s | 4 m 2 s | 90.0% |
| Full | 8,815 | 61,705 | 2 | 2 h 3 m 54 s | 2 h 5 m 44 s | 95.3% |
Hardware: two NVIDIA Tesla T4 cards (16 GB each) training in parallel, on Kaggle's free tier, which allows 30 GPU-hours a week. The experiment used 2.17 of them.[2] The trained checkpoint is 843 MB and runs on the Mac mini like the stock one.
Training took the free model from worse than a coin flip to within reach of the paid one.
The pilot alone — 300 repositories, three minutes of training — took it from 46% to 90%. The full run added five more points. It agreed with the paid model on 96.4% of a thousand repositories neither had trained on. It still trailed the paid model by 3.4 points on the exam, so it is not a replacement.
Trained, it kept three-quarters of the calls with no measured loss.
The trained model answers when its score is far enough from 50/50 and passes the rest to the paid model. The cutoff was chosen on the thousand unseen repositories, then applied unchanged to the exam.
| This job, per repository | Accuracy (exam) | Time | Cost per 1,000 |
|---|---|---|---|
| Jev alone | 98.7% | 280 ms median | $0.055 |
| Trained free model → Jev (free model keeps 73%) | 98.7% | ≈ 830 ms mean | $0.015 |
| Trained free model alone | 95.3% | 731 ms median | $0 |
| Frontier model alone (ceiling) | 97.3% | 3,790 ms median | $8.93 |
Why it is slower here than on the first tab
This job asks seven long questions per repository, and the trained model reads each one as its own pass of up to 1,024 tokens. That turned 50 milliseconds into about 730 on the Mac mini, more than twice the paid model's network round trip. A single-question version scored the same 95.3% on the exam; its speed has not been measured.
The thread
Worth training, not worth deploying on price — for this job. The reasons it might still be worth deploying are not about price.
The free tier earns its place when the tier behind it is expensive, or when price is not the constraint.
It does not pay in front of a cheap paid model. A 73% offload in front of a model that costs five and a half cents per thousand answers saves about four cents per thousand, and on a multi-question job it made each answer slower.
It can pay in front of an expensive model — but only trained. As shipped, it could keep about one call in eight before costing a point of accuracy. Trained on a job, it kept three in four in front of the paid decision model with no measured loss, which is the shape a frontier-model cascade needs. That combination was not tested here.
It pays when data cannot leave the machine. The paid decision model is an external service. The free one never sends a byte anywhere.
It pays when a vendor outage is unacceptable. A local tier that answers 95% of one job correctly keeps the decision running while the API is down.
It pays on single-question gates at volume, where it answers in about 50 milliseconds with no per-request cost — once it has been trained on that question.
Recalibrate before routing, train before trusting, and price the tier behind it first.
Never route on the confidence a model ships with. Fit a temperature on held-out data first. It costs nothing and turned a 64% "90% sure" into 89%.
Treat a stock checkpoint as untrained. Two hours of free GPU time moved one job from 46% to 95%. Judging the free model by its stock accuracy would have written it off.
Use the cheap paid model as the teacher. Its answers cost 54 cents for nearly ten thousand repositories and made a training set no one had to label by hand.
Direction
Before adding a free tier, price the tier it would stand in front of. If that tier costs pennies per thousand calls, the free one is a privacy or availability decision, not a cost one — and should be justified on those terms.
Same input to every model, answer keys no contestant wrote, thresholds chosen on separate data.
Head-to-head (300 questions). Fifty stratified test items each from six public, human-labeled
test sets: DAIR Emotion (6 options), SST-5 (5 levels), deepset prompt-injections (yes/no, English
and German), ANLI round 3 (3 options), Banking77 restricted to 20 fixed intents, and StrategyQA
(yes/no).[4] AG News and BoolQ were excluded because LAYA's documentation lists them in
its training mix. Seed 20260923. LAYA and Jev received byte-identical {state, questions} JSON;
Claude Fable 5.1 received the same JSON verbatim inside a fixed prompt, at high reasoning effort,
and returned a probability for every option. Zero malformed or refused responses on this set.
Calibration. Expected calibration error over ten equal bins, Brier score and log loss per model; the recalibration fits one temperature per question type with five-fold cross-validation, so no answer is recalibrated by a fit that saw it.
Cascade. Every tier's answers, latencies and costs were stored once, and every routing policy was simulated offline from them. The headline thresholds were chosen on four-fifths of the items and scored on the held-out fifth. "Not materially worse" was fixed at one accuracy point before any result was read.
Fine-tune. Jev (TypeSafe direct API, jev-1.13.0) answered seven yes/no questions for each of 9,815 repositories — $0.54, no errors. 8,815 became training data (61,705 question–answer pairs, the paid model's full probabilities as soft targets); 1,000 were held out to measure agreement and to choose the cascade cutoff. The 150-repository exam was never seen by any training step, and its labels were written blind by Claude Fable 5.1 before any model ran. Training: LAYA 0.3.10 from the English checkpoint, two epochs, LAYA's own reinforcement-learning script, two Tesla T4 GPUs on Kaggle, PyTorch 2.10 with CUDA 12.8.[2]
Hardware for inference. An M4 Mac mini with 16 GB of memory, macOS 26.4, Python 3.12, PyTorch 2.14 on Apple MPS. The GPU was about twice as fast as the CPU in a spot check (31 ms against 67 ms for one short question).
Spend. $3.57 for the head-to-head (Fable $3.55 billed by the gateway, Jev $0.011 estimated from tokens at list price), $0.54 for the training labels, $1.34 for a fresh Fable pass on the exam. $5.45 in total. The training GPUs cost nothing.
On the fine-tuning exam, one contestant wrote the answer key.
Build Log No. 005 recorded twice being caught scoring a model against labels it had written. The same trap was open here, and was sidestepped rather than avoided. The 150-repository exam's labels are Claude Fable 5.1's, so a fresh Fable run graded against them measures how well Fable agrees with itself. It is shown as a ceiling, not a competitor, and it scored 97.3% — including two empty replies, counted as wrong.
The agreement figure has the same shape from the other side. The trained model learned from the paid model's answers, so 96.4% agreement on unseen repositories says it copies its teacher well, not that either is right. The exam is the only number here that measures being right.
Proof-of-concept samples, one trained job, one machine
The samples are small. 300 mixed questions, and a 150-item exam where one question is 0.67 points. Treat gaps under about two points as noise.
One job was trained. Whether two GPU hours buys the same jump on support tickets, invoices or medical notes is exactly what one run cannot say.
The trained-tier cascade was tested in front of the cheap paid model only. In front of a frontier model it should matter more; that was not measured.
Latency is one machine and one network. The free model's times are on a single Mac mini with no other model loaded; the paid models' are from one laptop on one home connection. No load, no concurrency.
The package moves fast. Version 0.3.10 was published about a quarter of an hour before these runs began. A later checkpoint may behave differently on every number above.
- ExtLAYA, version 0.3.10, released Sep 23, 2026 — package pypi.org/project/laya, source github.com/NandhaKishorM/laya, checkpoints huggingface.co/convaiinnovations/laya (revision 5e7b2b1). Apache 2.0. Architecture, the three checkpoints, Route Mode, the yes/no label-bias warning and the fine-tuned benchmark figures are from the project's README. Quoted: "a fast base to specialise, not a zero-shot decision engine."
- ExtTraining: the project's own notebook,
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb, run as a Kaggle script on the free "GPU T4 ×2" accelerator. Kaggle's account quota readout on Sep 23, 2026: 30.00 GPU-hours per week, 2.17 used. - ExtPrices checked Sep 23, 2026 on OpenRouter: typesafe/jev-1.13 at $0.042 per million input tokens, output free; anthropic/claude-fable-5.1 at $10 per million input and $50 per million output tokens. Jev documentation: docs.typesafe.ai.
- ExtTest sets, all public test splits on Hugging Face: dair-ai/emotion, SetFit/sst5, deepset/prompt-injections, facebook/anli (test_r3), legacy-datasets/banking77, ChilleD/StrategyQA.
- DataExperiment files, Sep 23, 2026: the frozen 300-item set with a SHA-256 manifest; every raw response from every model, with latency, tokens and billed or estimated cost; the threshold sweeps for eight routing signals; the 9,815 teacher labels; both training runs' logs and timing records; and the scoring script that regenerates every figure on this page from those files without a new model call.