A decision model answers only the questions you write in advance, with odds attached, and cannot produce a word of prose. This is a working guide to one: the three kinds of question it takes, the jobs it fits, what to get right when you use it, and what a well-wired pipeline saves. Every point rests on a measured run on my own hardware — the runs are the evidence here, not the story.
Every model most people have used writes text back. This one cannot. You send it the thing to look at and the questions you wrote yourself, and it returns an answer to each one with a probability attached.
The model is Jev, from a company called TypeSafe. The answer can only ever be one of the options you supplied, so it cannot invent a category that isn't in your list, wander off topic, or pad a reply. What comes back is a number your own code can act on: keep it, drop it, or send it to something smarter.
The right picture isn't a consultant, it's a smoke detector. It isn't clever and it won't tell you what's burning. It is cheap enough to put in every room and leave running forever, and that is the whole of its value. One decision from it is worth nothing and ten million are worth a lot — so if the job doesn't have volume, this is the wrong tool.
Everything built with this model is built from these. Which one to use is decided by the shape of the answer you need, never by habit.
All three ride in the same request against the same item, and the item is read once. That makes one habit worth forming: ask the questions you might not need. Fourteen questions about one repository billed three hundredths of a cent — the same as two.
Write the rule down first. If the answer is yes, the job is a candidate. If someone would have to read, weigh, and decide, it is not — however similar the two jobs look.
The fourth card is the one to remember. Same model, same price, same corpus, same afternoon — and it lost to a chat model costing seven times more, because the question changed shape.
All four jobs read the same short repository descriptions, so difficulty and subject matter are not what separates them. The first three ask a question a written rule can settle. The fourth needs judgment.
Under other names the winning shape is everywhere — screening documents against inclusion criteria, routing a ticket to a queue, checking that a quoted passage supports the claim citing it, catching a prompt-injection attempt before an expensive model reads it.
Being slightly better than a frontier model on one gate is a curiosity. Being affordable at full coverage changes what you can build. Three cautions decide whether any of it reaches you.
On a job asked a few hundred times a month, saving 90% of a small number is a rounding error, and the pipeline costs more to build than it will ever save.
On the tagging job it lost every way it was wired — six attempts, the best at 75%, against a cheap chat model's 87% under the identical rulebook. What rescued that job was a pipeline in which this model narrows the work and something else decides it.
Nothing here was won by swapping a model. Every number came from writing rules, writing questions, and building a way to score the result. The model was the cheapest part of the work.
Don't ask a cheap model to be right. Ask it to be wrong in one direction only.
Choose the question type before you write the question. A yes/no, a pick-one and a rate-it are different tools, and the default is rarely the right one.
Rests on: the same tagging job scoring six points higher, at 40% of the price, once it was asked as a pick-one instead of 31 yes/nos.
Put it at the front of the pipeline, not at the end. Tuned to throw away 87% of a corpus it has not dropped a real hit in any measured run. Tuned to decide for itself it missed 24 of 212. Same model, one setting apart.
Rests on: the four jobs on the shape chart, and six failed attempts to make the fourth one work.
Spend the budget on the written rules and on scoring, not on the model. When a classification job looks subjective, it is usually under-specified.
Rests on: the best achievable score moving 80.0% → 95.8% → 98.7% across three rewrites of the rules, against a 1.9-point spread between the best and second-best model.
Before you believe an accuracy number, ask who wrote the answer key and how each model was called. This is the question to put to any team bringing you a benchmark.
Rests on: two numbers in one evening that flattered the model producing them — 96.3% becoming 81.5%, and a 99.4% agreement figure that was a model agreeing with itself.
A cheap decision model pays when it gets a narrow, explicit job inside a larger system — not when it is asked to imitate a general-purpose model. This tab is about the first half: making the job narrow and explicit. You can skim the numbers; the headings carry the advice.
The three types are easy to tell apart on paper and easy to ignore in practice. An audit of the first day's work counted every call: all sixteen used the yes/no form, including two jobs that were not yes/no questions.
Asked as 31 separate yes/nos, the options never compete against each other inside the model — they only get compared afterwards, in your code. Asked as one pick-one, they do. The rate-it type paid the same way on a filtering job: a described five-level ladder, added beside nine yes/nos, left 42% fewer items for the expensive reader with nothing missed.
The obvious move with 31 labels is to split them into 6 families, ask which family first, then which label inside it. The vendor publishes a cookbook for exactly this.
A hierarchy can never beat its own first step, and the first step is an abstraction. Asked for the family directly, the model did thirteen points worse than when the flat answer's own probabilities were simply added up into families. The leaf options carry concrete, tested conditions; the families were abstractions invented to group them — and a good abstraction is much harder to write than a good condition.
One piece of standing advice backfires if read loosely: "combine in code" means combining independent factors, not re-applying rules the options already state. Doing both cost four to five points every time.
Before scoring anything, find the ceiling: have a careful reader label a sample, then have an equally careful reader label it again, blind. Nothing can score above how often those two agree with each other.
A low ceiling is a bug in the rules, not a hard problem — and rewriting them is free.
The first rulebook had 7 rules and a low ceiling, which invites a comfortable conclusion — this task is just subjective. It was wrong. The rules were vague, and rewriting them twice took the ceiling nearly to the top. Same task, same data, same readers, 18.7 points, and not one model changed.
Five explicit exceptions were added to a rubric — carve-outs naming things to treat as out of scope. The judging model went hunting for reasons to call things out of scope, wrongly cleared about 33 items it should have cut, and on a dozen wrote a reason that contradicted its own verdict. One exception it applied had never been written down by anyone. A list of exceptions teaches a model to look for exceptions.
The threshold is a free dial. Because the answer is a probability saved to a file, trying forty cut-offs is arithmetic on numbers you already have. Asking a chat model to "be stricter" costs a full re-run every time.
Check the confidence before you route by it. Sending the unsure items to a person or a bigger model is the right pattern, but only where the confidence actually sorts right from wrong. On the pick-one type it did: right 93% of the time at high certainty, 32% at low. On the yes/no type, same job, it was flat — about as right when unsure as when certain — and routing by it would have sent the wrong items to the expensive reviewer.
Score once, on a fresh sample nobody has seen. The best-scoring design on the tuning sample was the single big question above. A team that stopped there would have shipped the worse design and had a better-looking number to show for it.
Narrow, explicit jobs were the first half. This tab is the second: the larger system around the model, where something that can actually read makes the final call.
Set the cut-off by what you can afford to miss, hand the survivors to something that can actually read, and keep those two numbers as separate settings in your code.
The worked example: this site publishes rankings of public code repositories and by policy does not name offensive-security tools. Ten thousand of them, refreshed constantly, and a judgment call on each one. The model returns a probability, and the obvious cut-off is the middle: above 0.5, call it an attack tool. That setting is the model deciding.
Those two numbers — the one that filters and the one that decides — were the same variable in the shipped script. A repository can only survive the filter by scoring above it, so almost every survivor read as a disagreement with the second model: 1,854 disagreements sent for expensive adjudication instead of about 70. Found by watching a live log during a production rehearsal, not by any test.
Three models from three companies, each one seeing only what the one before it could not settle.
Same catch, 73% less money. Both wirings find all 212 attack tools in 10,714 repositories. The difference is how many of them the expensive model had to read.
The same shape rescued the tagging job. The decision model filters out the non-AI half, a cheap chat model tags what's left, the decision model tags it again independently and flags every disagreement, and only the flagged items reach a careful third model. That lands at 89.6% for about $2.55 per thousand — against Claude Opus 5 working alone at 85.1% for roughly $5.77. Better and cheaper than the frontier model, and not because the cheap model is good at the job. It's good at narrowing it.
One honest note on the saving in the chart: that is a full sweep, and the hourly run only looks at what's new, where the saving is under a cent. The better argument for wiring it this way is headroom, not money.
When two models built on different principles disagree about an item, that item deserves a closer look. When two similar models agree, it proves much less.
The chat model made 20 errors in 154 items. Flagging every item where it and the decision model disagree flags 35% of the work and catches 15 of those 20. A second chat model as the checker caught 6, at eight times the price, because chat models share their mistakes — ten items fooled all three tried, identically. A model built on a different principle fails somewhere else, and that is the only property that makes a second opinion worth buying.
Asking it properly made it a worse checker. Switching to the correct pick-one question type made it agree with the chat model more often, so it caught 11 of 20 errors instead of 15. Where a component's job is to disagree, the clumsier wiring is the deliberate choice — optimizing every part for its own score can make the system worse.
And the expensive judge is not worth its premium. Twelve model-and-calling-style combinations were tried for the final step. Claude Sonnet 5 at $3.87 per thousand got 43 of 54 right; Claude Fable 5.1 at $15.93 got 44. One item, at four times the price — which at this sample size is a tie, not a ranking.
Two dozen numbered experiments against public code repositories and public research abstracts, each with its sample size, its random seed, its cost, and the file holding every individual answer.
The rule went into a file before any data was drawn. A frontier model labeled a randomly drawn sample blind, with no other model's answers in view. Thresholds were swept offline against saved probabilities. Then the design was frozen and scored once on a fresh sample the tuning had never seen — and that last figure is the one reported, except where a tuning-sample number is named as one.
Both times a headline figure was a model measured against something it had produced itself. Both times the number flattered it. Neither was caught by review.
The deeper problem was underneath it. The answer key for that job had been written by the same model working the same careful way, so 96.3% was never scored against an independent standard — it was scored against its own house style. That is self-consistency, not skill. The identical error had happened four hours earlier on the other job, where the answer key equaled the cheap model's own verdict on 99.4% of rows.
A contestant's output must never be the ruler. If the answer key and one of the models share an author, that model's score is not comparable to anyone else's.
A model's measured quality is a property of the call, not the model. A number taken from a careful hand-run pass cannot be compared to one from a production API call, even for the same model on the same day.
These are agreement figures, not accuracy. Every score is how often the model matched a reference that is itself a model.
The samples are small. A 2–3 point gap is noise: the same question asked twice on the same 154 items scored 73.4% and 72.7%, which is one item.
One corpus, one domain, one day. Whether any of it holds on insurance claims, support tickets or medical records is exactly what one day of work cannot say.
Speed for a live product is unmeasured. Everything ran in batches, so the number that decides whether this can sit inside something a person is waiting on was never taken.
It does not treat its input as hostile. Text written to steer a classifier can steer it, which matters the moment it reads anything a stranger wrote.
And it is not private. An external service, so anything sent to it leaves your machine — which ruled out several of the most attractive uses here before they were tested.
If you're evaluating this class of model: write your rule down, have two careful readers label 150 of your own items blind, and look at how often they agree with each other. That number tells you whether you have a model problem or a rulebook problem, and it costs nothing but an afternoon. Do it before the pilot, not after.