The Findings
Claims you’ve already heard, taken seriously enough to measure.
Every report starts with something you’ve probably run into already. A pitch, a headline, someone fresh back from a conference. The claim sounds plausible and the stakes are real money, so instead of arguing with it, I measure it.
Same shape every time. The claim, then the public evidence read against it. Every finding leads with its takeaway, carries a grade, and ends with what I’d actually do about it. The report gets attacked before it goes up, and when a finding doesn’t survive that, it says so instead of quietly disappearing.
- 11,000 people starred a fix for AI-looking design. I tested it. Looking like AI wasn't the problem.“If a page looks AI-made, it's slop, and fixing it means removing the AI look.”
- I heard software was dead. So I tested it. Here's how much was hype.“AI agents can rebuild any app now, so paying for software is over.”
- OpenAI launched a decision model. I tested it against the one I use on 3,887 questions.On five public tasks, each model on its own best wording, Jev from TypeSafe and OpenAI's Decisions API tied on plain labeling, and Jev was ahead on logic (73.7% vs. 65.1%). Rewording the questions moved both models up to 13 points, while the gap between vendors was under 4 points on four tasks of five. Jev was cheaper on every task and more honest about how sure it was. OpenAI was better with a long rulebook, gave the identical answer every time, and refused 5 items, 4 of them plainly harmless. On pictures (Revisions 2 and 5, the second with Jev 1.14 looking at the photos itself), the two filed photos into folders equally well, Jev was more honest about how sure it was and tagged photos for about 15% of OpenAI's cost, neither caught a visible mistake in an AI-made video, and Jev's new Box question found named objects in photos. Tags found photos far better than folders alone.
- Telling AI to think like Socrates: new voice, same answers, bigger billOn strong models, philosopher system prompts changed how the answers sounded and what they cost, not whether they were right. Every one of the 28 philosopher cases cost more than asking plainly, and no philosopher beat a same-length placebo prompt by more than the variation between two identical no-prompt runs. The one model that improved, a terse cheap one, improved under any long prompt.
- Letting AI agents tune the system: adopted fast, measured almost never“Leave the decision engine alone and put a team of AI agents above it, offline, to tune it between runs.”
- Buying an AI agent team: the manager isn't the expensive part“What you want is a manager agent directing worker agents across your tools and data, and the manager is the part worth paying up for.”
- What AI customer decisions should stand on“AI is going to make more of the decisions companies make about their customers, and you should get ahead of it.”