Telling AI to think like Socrates: new voice, same answers, bigger bill

In June a team at Argonne National Laboratory published a neat experiment.^[1] They gave AI models a system prompt written in the style of a famous philosopher (Socrates, Aristotle, Descartes, Kant, Hume, Hegel or Plato) and then asked them chemistry questions. The standout was Socrates. On one model, under the strictest scoring, it lifted the score from 51% to 73%. The gains were uneven, though, and the authors say plainly that no single philosopher won on every model.

That's the same idea behind the most common prompt advice out there... start with "act like an expert." So does telling a model who to be give it that person's thinking, or just their voice?

I wanted to know if it held up outside chemistry. So I took 15 bugs I'd already fixed in my own setup, where I know the right answer because I lived through it. I sent each one to four models, asked ten different ways, and graded all 600 answers blind.

The ten ways were: no system prompt (twice, to see how much a model disagrees with itself), the paper's seven philosopher prompts word for word, and a placebo. The placebo is a prompt the same length as a philosopher's, full of generic careful-reasoning advice, with no philosophy in it. If Socrates really helps, he should beat the placebo.

Here's how to read that. Each row is a model. The solid line is its score with no prompt, and the gray band is how far that score moved between two identical no-prompt runs. Circles are philosophers. The red dot is the placebo. On the two strong models, everything landed in the band.

So the scores held still. What changed was everything around them. Here's the same bug, the same model and the same settings, asked plainly and then asked as Descartes.

Both answers are right. One gets there in its first sentence. The other puts on a robe, clears its throat through three headings, and arrives at word 94.

Every score comes from a stored answer graded against a fix that's known to work, and every cost comes from the provider's own bill. Each of the 600 answers is saved with the exact request, the model's reply, its token count and what it cost.^[3]

Six predictions went into a written ledger before any answer came back, so they couldn't be quietly reworded afterward.^[7] Two of them were wrong. The tests below are listed whether they passed or not.

• A difficulty screenAll 25 candidate bugs went to all four models with no system prompt first. Seven that every model solved were dropped, since a prompt can't improve a perfect score. Three more with the least room were dropped too, leaving 15.^[4]
• An audit that changed the testIn the screen, Claude Sonnet 5 spent its entire 4,000-token allowance thinking on two bugs and returned nothing. That's a setting problem, not an answer, so the limit went to 8,000 for every model before the main run.
• Two unfair problems, fixedThe graders caught one bug that quietly assumed a tool was installed, and one answer key that expected facts the problem never gave. Both were fixed and re-checked before the main run.
• An audit that killed a resultThe first analysis said the noise was zero, because each model's two plain runs scored exactly the same. That made every difference look real. Looking bug by bug, two to four answers had flipped on every model, and the flips happened to cancel. The honest variation is about 2 points out of 15, and most of the differences disappeared into it. Two runs can show that a score moves on its own. They can't pin down how far.
• A null checkThe second plain run is a test where the right answer is "no difference." It came out that way in the totals, and the bug-by-bug flips inside it became the variation band you see in the charts.
• A grader checkGemini 3.8 Flash re-graded Sonnet's 150 answers on its own. Same verdict on 147, same right-versus-wrong call on all 150, and the totals were 140 vs. 139.
• Predictions that failedThe ledger predicted the weakest model would gain the most, and that Socrates would answer with questions instead of a diagnosis. Neither happened.

The calls cost $0.52 for the screen, $6.19 for the main run and $0.29 for the cross-check. The first draft took one morning, from reading about the paper to publication. No one ran an adversarial re-analysis, meaning a second analyst whose job is to break the result. That's listed with the other limits.

The calls went through OpenRouter, one provider pinned per model, with the paper's settings (temperature 0.2, top_p 0.3). The runner, grader batches and analysis are small Python scripts. The harness they grew from was built for an earlier prompt test in a separate session and isn't counted above.

Models, by role. Claude Opus 5.5 designed the test, turned my notes into problems, ran it, graded it and wrote this page. Gemini 3.8 Flash only re-graded. Claude Sonnet 5, GPT-5.6 Luna, DeepSeek V4 Flash and Gemini 2.5 Flash-Lite only answered. They run from a top-tier model to a 14-month-old budget one.

The prose of this report was written by an AI, Claude Opus 5.5, under my editorial direction. I own the question, the choice of bugs and every editorial call, including what changed in Revision 2.

The scores were graded by AI, blind, and checked by a second AI from a different company. The costs and token counts weren't produced by an AI at all. They're the providers' own billing records for each call.

What that means for you: you're trusting the stored answers, the answer keys and the billing records, and each of those is named in the citations.

• Rev 1 · Sep 27, 2026First publication.
• Rev 2 · Sep 27, 2026A polish pass. No result changed. "Normal wobble" became the variation between two identical no-prompt runs, since two runs can't measure a stable noise level. The cost-efficiency chart now sets each model's no-prompt result to 1.0. The plain-vs-Descartes example moved up to the Overview. The PDF now reads findings first and method last, and the analysis credit names the model.

Each bug was a real problem with a known fix, like a database query that reported 100% agreement for everyone, or a wait loop that started the next job while the first was still running. The model saw the symptom and the code and was asked for the root cause and the smallest fix. A grader marked each answer right, half right or wrong against the fix that actually worked.

Telling a model who to be changed how it talked. It didn't change what it knew.

1. On a strong model, skip the philosopher. You'll get the same answers for up to 2.3 times the cost. (1, 3)
2. If a cheap model answers too briefly and misses, ask it to show its work. Any long prompt did that here. (4)
3. Test a persona before you trust it. Run the same questions with and without it, and run the no-prompt version at least twice. Don't treat a difference smaller than the gap between those repeat runs as signal. (1, 2)
4. Judge an answer by its conclusion, not by how thoughtful it sounds. (5)
5. When you add long instructions to a thinking model, give it room and watch for empty replies. (6)

• Removed · Rev 1A philosopher leaderboard. The data can rank them, but every gap between philosophers on a model is inside the 2-point variation between that model's two no-prompt runs, so the ranking would be noise with names on it.
• Removed · Rev 1"Descartes and Hume doubled DeepSeek's score." True on paper, 3.5 to 7.5. But the placebo got most of the way there, and the same plain model scored 6.5 in the screen.
• Small. 15 bugs, four models, one run per prompt (two for the plain prompt).
• Two baseline runs. Two identical no-prompt runs per model show that scores move on their own. They can't show how far. The real run-to-run variation could be larger than 2 points, and DeepSeek's no-prompt score was 6.5 in the screen against 3.5 in the main run.
• No adversarial re-analysis. The audits in How it was tested were run by the same model that ran the test. Nobody independent tried to break the result.
• My own problems. The bugs come from my homelab. They're real, but they're one person's mix of databases, scripts and containers.
• Chemistry prompts, used unchanged. The philosopher prompts were written for chemistry. I used them word for word so I wasn't tuning them until they won. Prompts written for debugging might do better.
• Grading calls. On three bugs the graders accepted a fix that might not run exactly as written. Each call was applied the same way to all 40 answers for that bug, so it doesn't favor one prompt. On one bug the problem didn't say what the API returns later, so a reasonable guess was graded wrong against the real cause.
• One grader family. Claude graded everything. The cross-check covered the one model where that matters most, and agreed on 147 of 150.

1. ExtHarb, H., Sun, Y., Unal, M., Chia, N., Yang, Z., Ingram, B., Surendran Assary, R. "The ballad of LLM agents: philosophical reasoning for chemistry." Machine Learning: Science and Technology 7(3), 030503, Jun 17, 2026. Argonne National Laboratory. 243 ChemBench numeric questions; GPT-4o, GPT-5, GPT-5.1. iopscience.iop.org/article/10.1088/2632-2153/ae792d. Graded: peer-reviewed, a single lab, chemistry only. The authors state that no single philosopher won across every model and task type. Used here for the question and the prompts, not as a claim about debugging.
2. ExtThe paper's code and prompts: github.com/HassanHarb92/Sci_reasoning_LLMs. The seven philosopher prompts (sys_prompts/*.txt) were copied unchanged at commit 0682e819. The comparison prompts are in the repo's baseline test script (run_one_control_async.py). Graded: primary source.
3. DataThe run: 600 calls through OpenRouter on Sep 27, 2026, each stored with its full request, reply, provider, token counts and billed cost (results.jsonl, written by run.py; order shuffled with a recorded seed; one provider pinned per model, no fallbacks).
4. DataThe problems: 25 candidate bugs drawn from the author's own notes on problems already solved, each with an answer key and a note of where it came from (PROBLEMS.md); 15 kept by a no-prompt difficulty screen (SCREEN.md, 100 calls, $0.52). Hostnames and addresses removed.
5. DataGrading: one blind grader per bug, 40 shuffled answers each, no model or prompt names attached, checked for name leaks before grading (grade_prep.py). Scores and the variation band are computed by analyze.py. The band is the bug-by-bug change between two identical no-prompt runs, an observed variation from two runs, not a measured noise distribution.
6. DataCross-check: Gemini 3.8 Flash re-graded all 150 Claude Sonnet 5 answers with the same rubric (crosscheck.py, $0.29). Exact agreement 147 of 150.
7. DataPredictions: six hypotheses written into the research ledger before the main run, with the method attached, then settled after it. Two supported, one mixed, one untestable at this size, two refuted.