OpenAI launched a decision model. I tested it against the one I use on 3,887 questions.

A decision model is a small AI that only answers closed questions. Is this email a complaint, yes or no? Which of these six feelings is this sentence? Rate this review from one to five. It doesn't write anything. It picks an answer and tells you how sure it is, as a number. That number is the useful part, because you can set your own cutoff after the fact: send everything under 80% sure to a person, for example.

For the last few weeks I've used Jev, a decision model from TypeSafe, to sort and filter the research that feeds this site.^[1] On Oct 6, OpenAI opened its own, called the Decisions API, in public beta.^[2] Same three kinds of question, a similar price. They aren't the only two (open-source ones exist), but they're the two I could put side by side on the same day.

So I ran them against each other twice on Oct 9: five public datasets where people had already graded the right answer, 3,887 test questions, and the same budget for each model to find its best wording before the test. On Oct 12 I added three small picture tests, because OpenAI's model reads pictures and Jev didn't. Two days later TypeSafe released Jev 1.14, which does, so on Oct 20 I ran every picture test again with Jev looking at the photos itself, and the text test too.

Here's what you'd likely assume about two decision models from two vendors, and what the test said about each. One held up, two didn't, and three held only halfway.

The answer in one breath

The first round was quick and gave Jev a home-field advantage: questions tuned on Jev were copied to OpenAI word for word. So I threw out its accuracy numbers and ran a fair one the same afternoon. The rules for that round were written down and saved before any practice question went out, and the test ran once.^[3]

Each model got the same four tries per task to find its best wording: one shared starting wording, one written from that vendor's own documentation, one in a different shape, and one revision written after reading up to 20 of that model's practice mistakes. The highest practice score won, and the picks were frozen before the test set was touched.

On Oct 20 Jev 1.14 went through the same steps: the same four wordings on the practice questions, no new one written for it, picks frozen, then the test once. It landed where 1.13 had, 71.3% overall against 71.1%. Parts A to D report the Oct 9 round, with the 1.14 numbers in each finding's notes.^[17]

Claude Haiku 5.5 ran alongside on Oct 12 as a yardstick, because it reads pictures at about the same price.^[8] OpenAI's and Haiku's answers are from that day; Jev 1.14 answered the same questions about the same photos on Oct 20.^[17] TypeSafe labels Jev's picture features a preview, not yet up to its standard for production, so Jev's picture results below may change with the next release.^[16]

f10open1 video, 2 visible mistakes

Neither model caught a visible mistake in an AI-made video.

MeasuredOn frames where a pencil floats in mid-air, OpenAI put "is anything floating?" at 0.00 and 0.02, and Jev at 0.10. They failed in opposite ways: OpenAI called 8 of 264 frames an AI mistake, Jev 201.

Measured in the testFrom the stored calls

floating pencil · OpenAI 0.00 · Jev 0.10Jev's box for “the floating object”its box for “the pencil”4:16, a real mistake4× zoom

The last 40 seconds, one bar per frame: how likely each model said the answer was yes.

Is anything floating?

OpenAI

Jev

Is any object duplicated?

OpenAI

Jev

3:444:23

The whole video: frames each model called an AI mistake, of 264

OpenAI8Jev201

• A mistake Rick had marked
• OpenAI
• Jev 1.14

Show the numbers

Frame	What	Question	OpenAI	Jev
4:16	pencil floating in mid-air	floating object	0.00	0.10
4:17	pencil floating in mid-air	floating object	0.02	0.10
4:15	pencil floating, as Earl dissolves	floating object	0.37	0.44
3:53	second pencil in Earl's hand (one behind his ear)	duplicate object	0.01	0.19
4:19	end-card logo, floats on purpose	floating object	0.96	0.91
4:16, zoomed in	the floating pencil, cropped to fill the picture	floating object	0.15	0.17

264 frames, one per second of a 4:23 draft, six questions each: 5.6¢ on OpenAI (Oct 12), 3.4¢ on Jev 1.14 (Oct 20).

For whoever picks the model: Don't count on a decision model to catch mistakes in AI-made video. This is one video, so treat it as a first look.

Evidence and objections

• Measured Rick makes short videos with AI video tools, and the clips come back with mistakes. A prop drifts, a second pencil appears, an arm vanishes behind a clamp. He'd already marked those by hand while reviewing one of his own videos. So the test sent one frame per second of that draft, each with six yes-or-no questions. It cost 6 cents.^[9]
• Measured Two of the marked mistakes were plainly visible in the sampled frames, the floating pencil and a second pencil in Earl's hand. OpenAI caught neither. Its highest scores went to motion graphics doing their job: labels halfway through a fade, a counter mid-roll, the end card. Cropping in so the pencil filled the picture only raised its answer to 0.15.
• Measured Jev 1.14 missed both too. It put the floating pencil at 0.10, 0.10 and 0.44, the second pencil at 0.19, and the cropped pencil at 0.17. Where OpenAI said almost nothing was wrong, Jev said almost everything was: "an AI mistake" on 201 frames and "a hand error" on 70. Its top flags were crossfades, two Earls and two moons halfway through a dissolve. That's real in a single frame and intended in the video.^[17]
• Measured Jev's new Box question (f14) was asked where things were. At 3:53 it boxed the pencil in Earl's hand. At 4:16, asked for "the object that is floating in mid-air," it boxed the moon.
• Measured Claude Haiku did no better. It missed the same two mistakes, flagged the same logo, and called the pencil in Earl's hand a multimeter. OpenAI's own guide warns that its vision models give approximate counts and struggle to place things precisely.^[10]
• Objection, stated: one video with two visible mistakes shows a blind spot. It doesn't measure how often one happens.

None of these picks a vendor for you.

1. For plain labeling at volume, either works.

   Jev was cheaper on every task.

   From f1, f6

2. If the question takes a step of reasoning, test Jev first.

   It was the one gap in the text tests that wasn't luck.

   From f2

3. Before switching vendors to fix accuracy, read 20 mistakes and reword the question once.

   Both models shared the same blind spots, and the wording was the part I could change.

   From f3, f4

4. If you'll set your own cutoffs, count the answers at exactly 100% first.

   A pile of 100%s leaves the dial nothing to rank.

   From f5

5. On OpenAI, ask one good question per item.

   It bills the item again for every question.

   From f6

6. If the decision needs a long rulebook or an identical answer every time, test OpenAI first.

   Lock the wording and the option order too.

   From f7, f8

7. Wherever you use OpenAI, handle a refusal as "send to a person."

   Never as "no."

   From f9

8. To sort photos, use a few folders a photo can't belong to two of.

   Either model files well. Let the confidence number decide which ones file themselves; Jev's was the more honest one.

   From f11, f12

9. To find photos later, tag them, and have Jev look for the first pass.

   If you'll keep adding tags, save a description of each photo too.

   From f13

10. To find where something is in a photo, try Jev's Box beside a yes-or-no.

    Keep only the boxes it's at least half sure of.

    From f14

11. Don't rely on a decision model to catch mistakes in AI-made video.

    OpenAI and Claude Haiku missed them, and Jev flagged nearly every frame.

    From f10

Say there's a folder of 2,000 mixed photos, screenshots and family pictures, and the goal is to find anything in it later. Here's the plan I'd follow, built from f10 through f14.

Plain code handles everything it can for free, because date and place are most of how people look for a photo ("the beach trip in 2023"). Jev looks at each photo once for a folder and its tags, about 20 cents for all 2,000. OpenAI picks a folder from the same photo, and any photo they disagree on, or that Jev isn't sure of, goes to a review pile. Two independent opinions catch more mistakes than one. If the tags will keep changing, have Claude Haiku describe each photo once and save the text, so a tag thought of next year re-reads words instead of pictures.

• That one vendor is better overall.
• That the picture results hold for screenshots, documents or family photos, since every photo here was a scene like a kitchen or a street.
• That Jev's picture features are ready for production. TypeSafe calls them a preview, and their accuracy can change between releases.
• That one video with two visible mistakes says how often a vision model misses them.
• That these results will hold after OpenAI's beta becomes a full release, since it's one model in beta.
• That they hold for tasks, languages or wordings I didn't test.
• That these are the only two decision models worth looking at.

Three things were written down and saved before any result came back: the fair round's rules, committed in three steps on Oct 9; Earl's prediction for the video test, written into its plan on Oct 12; and the Jev 1.14 re-test's rules with six predictions, committed on Oct 20 before its first test call. Here's how each one came out.

Oct 20: the Jev 1.14 re-test

Not predicted: Jev looking at photos came out cheaper than the describe-then-read route it replaced, and its video failure was the opposite of OpenAI's.

The fair round exists because the first round failed a fairness check. Here's everything that was checked.

Removed before Revision 1: the first round's headline, "Jev is more accurate," and its accuracy percentages. Its cost, repeatability, option-order, rulebook and refusal findings stand, because the fair round didn't test them and doesn't weaken them.

Standing limits:

• One question-writer. Claude Opus 5.5 wrote both models' questions. The equal budget and the published practice scores reduce that risk without removing it.
• One run of the test. Each model's picked wording ran once on the test set. The 95% ranges in the charts show sampling luck, not run-to-run wobble.
• Public datasets. These datasets are years old and widely published, so either model may have seen some of them in training. That would flatter both, not necessarily equally.
• A beta. OpenAI's Decisions API is in public beta with one model, gpt-6-luna. Its prices and behavior can change at full release.
• A preview. Jev's picture input and Box are labeled a preview, not ready for production, and every call needs a special switch turned on. Their accuracy can change between releases.
• Two dates. OpenAI and Claude Haiku answered the picture tests on Oct 12, Jev 1.14 on Oct 20, on the same photos and questions.
• Box on one dataset. f14 rests on 100 everyday objects, each present once. Crowded scenes, small objects and people weren't tested.
• One video. f10 rests on one of Rick's videos, with two mistakes visible in the sampled frames. It shows a blind spot. It doesn't measure how often one happens.
• Scene photos only. Every photo in f11 to f13 shows a place: a kitchen, a beach, a street. Screenshots, documents and photos of people weren't tested.
• A rule decides what a tag means. In f13 a tag counted as present when its outlines covered at least 0.5% of the photo. A different rule would move the tag numbers.
• Small sets. 100 to 328 photos or objects per test. The 95% ranges are about plus or minus five points, so gaps smaller than that are called ties here.
• Some keys are AI-built. Two first-round tests (the 31-category sort and the paper filter) were graded against keys from blind AI passes, and one leaned on Jev's own earlier reading. None of them backs a headline accuracy claim.
• One laptop, one day. Speed was measured from a home connection on Oct 9. Treat the speed result as a rough tie.

Both text rounds ran on Oct 9 between about noon and 2:20 PM ET, from a laptop, through a small script that calls both APIs and returns their answers in one shape.^[14] Every call was saved with its request, its answer, its token count and what it cost, and loaded into a database with a dashboard over it, so the numbers here can be re-cut without re-running anything.^[5]

The first round cost $0.90, the fair round $0.91, and the setup probes about $0.05. The prompt counts come from the session transcripts. Drafting the report isn't counted.

The Jev 1.14 re-test ran on Oct 20 between about 12:50 and 1:20 PM ET: the text test again plus six picture tests, $1.21 in all, every dollar of it on Jev. OpenAI's and Haiku's earlier answers were reused, not bought again, since neither model had changed.^[17]

The video at the top was made with the video kit used for every explainer on this site.^[15] Claude wrote and animated it in code, voiced it with ElevenLabs, and Rick reviewed it on a page that pins his notes to moments in the video. The first cut was finished Oct 9. On Oct 12 it gained the picture tests and grew to 3 minutes 39 seconds, after 16 review rounds and about 2 hours 40 minutes of build time. Up to that cut, its only costs were 278 voice credits and 2 cents to check the voice track against the script. Revision 5's cut adds Jev 1.14 looking at the photos and its Box question. The music and sound effects came from a library Rick already had.

Models, by role. Claude Opus 5.5 designed both rounds, wrote every question for both models, ran them, scored them, built the video and wrote this page. Jev 1.13 and OpenAI's gpt-6-luna only answered. Fable 5.1 built two of the first round's answer keys blind, and those keys aren't used for any headline accuracy number. In Revision 2, Claude Haiku 5.5 answered the picture questions as a reference and wrote the photo descriptions Jev read. Claude Opus 5.5 designed and scored the picture tests and drew the figures. In Revision 5, Jev 1.14 only answered; Claude Opus 5.5 designed, ran and scored the re-test and redrew the picture figures.

The prose here was written by an AI, Earl (Claude Opus 5.5), under Rick's editorial direction. Rick owns the question, the decision to throw out the first round and every editorial call.

The accuracy numbers weren't produced by an AI judging anything, with two exceptions. They're plain code comparing each answer to a key that people wrote for public datasets, and the costs come from each API's own billing count. f4's three patterns are Claude's reading of each model's practice mistakes. And in f10, whether Jev's boxes landed on the pencil was Claude looking at the boxes drawn on the frames.

So you're trusting the public answer keys, the stored calls and the locked rules, and each of those is named under Sources.

Rev 1 Oct 10, 2026: first publication.

Rev 2 Oct 12, 2026: adds pictures. Part E has four new findings: a missed mistake in an AI-made video (f10), filing photos into folders (f11), folder design against rewording (f12), and finding photos with tags (f13). A box under the answer summed them up, f6 notes that pictures aren't billed again per question, and the conclusions added three recommendations and a plan for organizing a pile of photos. The limits cover the picture tests. No text result changed.

Rev 3 Oct 12, 2026: the video at the top is the new cut that covers the picture tests (3 minutes 39 seconds). How it was made counts the video's second round of work. No finding changed.

Rev 4 Oct 15, 2026: moved to the notebook design. The story comes first, every finding sits beside a figure drawn for it, and each statement is marked as measured or worked out. "How I checked" now opens with the rules and the prediction written down before the tests. f4 now says its patterns are Claude's reading of the mistakes, not a reading by hand. Every figure is now rebuilt from the stored runs, with each number checked against the text, and that check corrected six details. None changes a conclusion. f9 had called all five refused items harmless, but one is a trivia question about suicide. The trivia lead in f1 is 3.9 points, not 3.8, and OpenAI's trivia gain in f3 is 8.3 points, not 8.4. f12's fresh photos number 240, not 237. f13 averaged 22 searches, not 21, and the describe-once route cost 42% of OpenAI's, not 40%. The fair round made 19,774 calls, not 20,560. The 0.21-second speed tie is now credited to the first round's speed test, where it was measured.

Rev 5 Oct 20, 2026: TypeSafe released Jev 1.14 on Oct 14 with picture input and a new Box question, so every picture test ran again with Jev looking at the photos, plus a new test of Box (f14). f10, f11 and f13 now compare Jev with OpenAI directly. f11 adds that Jev was more honest about being sure, f12 adds folders defined by example photos, and f13's cost finding flipped: Jev looking costs about 15% of OpenAI and less than the description route. The playbook, the photo plan and the limits changed with them. The text test ran again on 1.14 and held (71.3% against 71.1%), so Parts A to D keep the Oct 9 numbers with the 1.14 results in their notes. The video at the top is a new cut that covers Jev 1.14.

1. SourceTypeSafe. Jev documentation, including the API reference and the Jev 1.13 model notes on where it is weak, read Oct 9, 2026. Graded: vendor documentation, used for how the product works and how its questions should be written, not for any claim about accuracy. docs.typesafe.ai/llms.txtjev-docs
2. SourceOpenAI. "Decisions" guide and API reference (the create method, developers.openai.com/api/reference/resources/decisions/methods/create), public beta since Oct 6, 2026, read Oct 9, 2026. Graded: vendor documentation, used for how the product works and how its questions should be written. Where the guide and the reference disagreed (hosted image links), a direct test settled it. developers.openai.com/api/docs/guides/decisionsopenai-decisions-docs
3. DerivedThe fair round's protocol, written and committed before any practice call (fair/PROTOCOL.md, 2:07 PM ET Oct 9, 2026), the practice results and revisions (2:13 PM), the frozen picks (frozen.json, 2:14 PM), and every candidate wording with its practice score (fair/results/practice.json, candidates.py).dc-protocol
4. SourceThe five public datasets, all used through Hugging Face with their published answer keys: StrategyQA (ChilleD/StrategyQA, yes or no with multi-step reasoning), Adversarial NLI round 3 (facebook/anli, written by people to fool models), Emotion (dair-ai/emotion, six feelings), SST-5 (SetFit/sst5, five-level movie review ratings) and Banking77 (legacy-datasets/banking77, 77 banking requests). 300 items per task held for practice, the rest for test, and no item from earlier tests on this site. Graded: established public benchmarks with people-written keys.five-datasets
5. DerivedThe fair round's test: 3,887 items, each model's frozen pick run once, every call stored with request, answer, billed tokens, cost and latency (fair/raw/test.jsonl), scored by fair/score.py into fair/results/test.json: accuracy with 95% ranges, exact McNemar tests per task and pooled, calibration error before and after a one-temperature fit on practice, share of answers at exactly 0 or 1, refusals, cost per 1,000. Every figure's numbers are rebuilt from these files by notebook/export_notebook.py, which checks each against the text.dc-test
6. DerivedThe first round: 6,468 scored calls and 640 speed calls on Oct 9, 2026, $0.90, in four tests (a 300-item people-graded benchmark, a 1,246-project hacking-tool filter, a 154-project 31-category sort, a 118-paper research filter), stored per call (raw/*.jsonl) and scored offline (score_{a,b,c,d}.py, results/arm_{a,b,c,d}.json, FINDINGS.md). Its accuracy claims are superseded by the fair round, and the file carries that banner.dc-round1
7. DerivedBilling probes: the same short support ticket asked 1, 5, 20 and 100 questions on each API, with the billed tokens each returned (DESIGN.md, probe records p01, p02 and p05).dc-billing-probes
8. DerivedThe picture tests, Oct 12, 2026: the experiment plan with its prediction written before any test ran, four billing probes on video frames, and one folder per test with its raw calls, scoring script and results (visual-linter/PROPOSAL.md, probes/, run-01-earl-v4/, run-02-photo-filing/, run-03-tags/). Total spend $1.21.dc-pictures
9. DerivedThe video test: one frame per second of a 4 minute 23 second draft of one of Rick's videos, 264 frames, six yes-or-no questions each, $0.056 (run-01-earl-v4/results.jsonl). The answer key is Rick's own review notes on that draft, written between 10:50 PM Oct 11 and 1:49 AM Oct 12, before the test was planned. Follow-ups: a zoomed crop, and Claude Haiku 5.5 on the same frames (followup.json).dc-video-test
10. SourceOpenAI. "Images and vision" guide, limitations section (counting and spatial reasoning), read Oct 12, 2026. Graded: vendor documentation, used for what the vendor itself says its vision models do poorly. developers.openai.com/api/docs/guides/images-visionopenai-vision-docs
11. SourcePlaces365 (Zhou and others, MIT), validation set: photos of 365 scene types, each labeled by people. Used through Hugging Face with the official category list (CSAILVision/places365). 237 photos in 12 folders and 298 in 60, then 540 fresh photos (240 and 300) for the rewording test. Graded: an established public benchmark with a people-made key.places365
12. DerivedThe photo tests: every call stored with its answer, confidence, billed tokens and time (run-02-photo-filing/results-*.jsonl, variants-*.jsonl, run-03-tags/results-*.jsonl), scored by score.py in each folder. The rewording tweaks and folder descriptions (folders.json) were frozen before the fresh photos ran.dc-photo-tests
13. SourceADE20K (Zhou and others, MIT), validation set: scene photos where people labeled the scene and outlined every object. Used through Hugging Face. 328 photos in 8 scene folders, with each tag counted as present when its outlines cover at least 0.5% of the photo (run-03-tags/build.py). Graded: an established public benchmark with a people-made key.ade20k
14. DerivedThe shared client that calls both APIs and returns one answer shape, and translates questions between their formats mechanically (clients.py), plus the 37 mechanics probes run before either round (probes/*.jsonl, DESIGN.md).dc-client
15. DerivedThe video: a video-kit project (studio/projects/jev-vs-openai), with its review log of 16 rounds and the spend ledger (278 ElevenLabs credits, $0.02 of transcription). The first cuts were approved Oct 9, 2026, and the current cuts, which add the picture tests, on Oct 12.dc-video
16. SourceTypeSafe. "Images", "Box" and "Models" pages of the Jev documentation, read Oct 20, 2026: Jev 1.14 (released Oct 14) takes pictures and answers Box questions, both labeled "Dreamer-ready," a preview not yet up to the company's standard for production; pictures are billed as input tokens at the text price; sizes from 280 to 1,568 pixels a side. Graded: vendor documentation, used for what the product does and its stated limits, not for any claim about accuracy. docs.typesafe.ai/images.mdjev-images-docs
17. DerivedThe Jev 1.14 re-test, Oct 20, 2026: rules and six predictions committed before the first test call (jev-1.14/PROTOCOL.md), text picks frozen before the test set ran (t1/frozen.json), every call stored with its answer, confidence, billed tokens and time (t1/ to t6/), scored by score_photos.py, score_tags.py and the fair round's own scorer. OpenAI and Claude Haiku answers are their stored Oct 9 and Oct 12 runs on the same items. Total spend $1.21, all on Jev.dc-jev114