Buying an AI agent team: the manager isn't the expensive part

That distinction sounds academic until it starts costing money — because a company that believes AI is the strategy will buy the most impressive version of it, and the most impressive version is not the one that pays.

The vision currently being sold to executives is a team of agents: one "orchestrator" directing several "workers," each sitting on top of company tools and data, doing the routine work of a department. An agent, plainly, is a program that uses a language model to do a multi-step job — read a ticket, look something up, make a change, check its own work — rather than answering one question and stopping. The orchestrator is the agent that decides which other agent or tool handles each step, and when the job is finished: the shift supervisor, not the smartest person on the floor.

This report does not argue against that picture. The evidence broadly supports it. The question it asks is narrower and more useful: if teams of agents really are the coming shape of work, what is the least expensive way to get there, and how would a company know whether it was working?

Most executives have been shown a version of this slide, or drawn one:

Nothing about it is wrong. The difficulty is that the picture is silent on four decisions, and those four decisions are where essentially all of the cost, all of the compounding, and all of the exposure are determined. Same diagram, with the silences marked:

A company can implement that first diagram four different ways and see a twenty-fold difference in running cost, with one version improving every month and another staying flat forever. The rest of this report is about those four decisions — the architecture the evidence points to is drawn in Conclusions. Four positions come out of the evidence.

Hypotheses were written down before they were checked, together with the method that would settle them; derived claims are stored with the exact query behind them, so any number can be re-run by anyone with the corpus.^[1] Of the 22 claims on the ledger, 11 were the author's own hypotheses — and most did not survive intact: 14 claims settled as mixed, one was outright refuted and became this report's headline finding, one proved untestable and says so.

• Adversarially re-analyzed, second pass — a fresh session sent to destroy this reportBefore publication, a red team with no stake in the draft attacked every finding: all 18 cited papers re-read at the methods level, every quoted number re-derived or traced to its source, the classifier re-judged with a third model family, the collection censoring tested for time correlation. No finding died. Five were reframed: the orchestrator comparison recut around the same-toolbox baseline (F7), the social-corpus levels re-anchored to the reference judge (F3), the decision-path cost ground narrowed to the frontier tier (F19), the bandit finding scoped to offline experiments (F20), and one finding's numbers replaced because their original query had never been persisted (F18).^[6]
• Caught by that red team: the trust tile itself was wrongThe "sources read at full length" count was rendering the instrument's whole library — including 3,154 video transcripts belonging to a different sensor — instead of the ~65 documents actually read for this report. Provenance inflation of roughly fifty-fold, on the page whose only job is trust, live until the red team read the query behind the tile. Fixed by scoping the count; kept here because a trust page that has been wrong should say so.^[6]
• Caught by that red team: the AI disclosure was falseAn earlier revision of this tab said "no model scores, ranks or classifies any evidence." Untrue: the social-corpus composition is produced by a model classifying posts. The disclosure below now states what actually happens and how the classifier's error was measured, instead of denying it exists.^[6]
• Killed: the flagship experimentThe plan was to rent a GPU and test whether filtering training data through a verification gate beats training on everything. A corpus search found the decisive version already published, with controls that would not have been designed here — including one that swapped only the gate and watched the entire gain vanish. The experiment was canceled rather than run, and the finding now cites someone else.^[14]
• Killed · Rev 4: a second experiment, for the same reason as the firstA bake-off was designed to settle this report's own headline — swap only the supervisor across four price tiers over a fixed worker pool, on a battery of knowledge-work tasks, with a gate per task and cost measured per passing result. It was budgeted and about to be built. A corpus and literature sweep run before spending found the controlled version already published by an academic group, with twelve models instead of four, execution-based scoring instead of a model judge, and a design detail nobody here had thought of: crippling the supervisor's own perception so that delegating is forced rather than optional.^[30] The experiment was canceled and F24 cites someone else. This is the second time on this report that searching first replaced building — and the first time the thing it replaced was an experiment intended to test the report's own conclusion.
• Caught · Rev 4: this report's own reading of a source it already citedThe orchestration finding had been carried through three revisions on a summary of its source rather than the source's results table. Re-mining the full text for this revision showed the central sentence generalized from one model to all frontier models, when the table shows two others moving the opposite way. No red team caught this; no external party challenged it. It was found by re-reading something already on the shelf, which is the argument for re-deriving rather than re-checking — and it is the correction described in Removed & limits.
• Recounted · Rev 4: the source tile, againThe full-text shelf that this report's "sources read" tile counts has since been bulk-loaded with roughly 1,300 further documents belonging to other work. Counting the table would have restated 65 as a four-figure number — the same failure mode the red team caught at 50× in Revision 3, arriving by a different route. The tile now counts the curated shelf for this report (68: 43 papers, 25 web documents) and says so in its own label. A provenance number is not safe once it has been audited; it is safe while the thing underneath it holds still.
• Found and repaired · Rev 4: the shelf recorded where its sources were stored, and it was wrongSourcing the new citations turned up a defect in the instrument's own provenance. Of the arXiv documents on the full-text shelf, 1,244 recorded a storage location that does not exist — a collector running inside a container had written the container's private view of the path into the database. The files themselves were never at risk: every one was found, and a hashed sample matched its stored checksum exactly. But "stored as an immutable original with a content hash" is a trust claim this report makes on its citation page, and for those rows the pointer behind it led nowhere. The same collector was also storing each paper's title as the raw query string used to fetch it, because a pattern written to read the paper's title was reading the search feed's title instead. Both were traced to one collector, fixed at the source so the next run cannot repeat them, and backfilled: 1,244 titles and 1,244 paths corrected, no file missing, a ten-row sample re-verified byte-for-byte against its recorded hash. A provenance chain is only as good as the weakest pointer in it, and nothing about this one looked broken — the rows returned data, the hashes were right, and the titles were merely ugly rather than absent.
• Killed: a novelty claim"Provisioning an agent as a scoped bundle of capability and permission" was held as the novel half of the argument. A sweep found a July 2026 paper opening with the same problem statement, with a released dataset and a stronger principle. The novelty claim was dropped.^[19]
• Reversed: a design rule the author believed"Weights hold behavior, files hold facts" was proposed as a clean split. A controlled comparison found the opposite on accuracy — tuning beat retrieval by nearly seven points while retrieval added nothing. The rule survives as a risk argument, not a capability one.^[18]
• Reversed: the architectural assumptionThat a loop should be supervised by a frontier model was contradicted by a measured result in the other direction. Recorded as refuted on the ledger, and it became the headline of this report.^[10]
• Null check: is routing research just tracking the archive's growth?Routing paper counts rose steeply — but the archive itself nearly tripled over the same window. Normalizing to share of corpus was built so that "no real trend" was an available answer. The share rose anyway, roughly fifteen-fold. The keyword remains uncleaned, so the shape is published and the level is not.^[2]
• Null check: is the filings trend just a change in who files?The rise in control-layer language was re-run on a balanced panel — only companies filing in all four years — so that composition change could produce a flat line if it were the true cause. The trend got stronger, not weaker.^[3]
• Failed, then rebuilt: the practitioner-sentiment sensorA social corpus of 4,328 posts was expected to show what developers say about these tools. It contained zero matching posts — it had been collected scoped to named projects, so the silence was an instrument gap, not a result. A topic-scoped collector was written and run: 18,620 posts, 15 terms, no errors.^[4]
• Caught: keyword precision was NOT flatThe new corpus was audited by month before any finding rested on it, on the assumption that flat precision would license the trend. Precision was not flat — it rose from 67.5% to 75.5% as the vocabulary settled, which would have manufactured part of any trend measured on raw counts. Every composition figure was recomputed within confirmed on-topic posts.^[5]
• Failed: a trend read off a sample too small to carry itAn early pass called advocacy "flat" from the 60-per-month reference sample. At that size the noise band is about ±6 points — wider than every trend in the finding. Re-measured at 400–800 posts per month, advocacy rises modestly. The error was in reading a trend off a sample sized for accuracy, not for power.^[5]
• Measured the judge before trusting it — then got caught quoting the wrong judge anywayThe reference model was run twice over the same 60 posts at temperature zero to establish a ceiling — 98% self-agreement on topic, 97% on who was speaking. Candidates were scored against that ceiling, and the cheap winner was weak on identifying the speaker (68% against a 97% ceiling). Revision 2 then published the cheap model's speaker levels anyway, while its provenance text claimed the reference model's. The red team caught it: the cheap model runs 6–10 points hot on the practitioner share in both years. This revision publishes the reference model's stratified-sample levels, which is what this paragraph should have described the first time.^[5][6]
• Failed: the assumption that a prior bake-off transfersThe model that won an earlier classification job on corporate filings came fourth here, and the winner on this task was a different one entirely. Neither price tier nor past performance predicted accuracy. Whole-corpus classification cost $0.31.^[5]
• Defeated by its own cost control: the share measurementFive control terms were collected specifically so topic volume could be read as a share of conversation rather than a raw count. Collection was then capped at four queries per month to fit a time budget — and that cap pins the busiest terms at a ceiling of roughly 160 posts per month. Every control term and the three busiest topic terms sit at that ceiling, so both sides of the ratio are clipped and the share cannot be computed. The controls were collected and are unusable for their stated purpose. Composition measures, being ratios within a month, are unaffected — and the red team verified the 22 censored windows split 12/10 across the two years, too small and too balanced to manufacture the composition trend.^[4][6]
• Killed by the prior red team — and nearly missed twiceA within-company panel finding (F22) was carried in from earlier work, re-run, and confirmed to reproduce numerically. It was already dead: an adversarial re-analysis had shown the qualifying companies are mostly sellers of the software in question, that two are outright false positives, and that a placebo panel puts the persistence rate between on-premise and blockchain. Re-running the query verified the arithmetic and missed the critique entirely. The finding is kept in its broken state rather than deleted — and the second red team then found the same dead reading reused, un-caveated, in a different claim's evidence on the ledger. It is annotated there now.^[27][6]
• Excluded: a widely quoted industry statisticA frequently cited figure on vendor purchases outperforming internal builds was ruled inadmissible by the prior red team — a small interview-and-survey base, publicly criticized methodology, and unpublished underlying data. It is not cited anywhere in this report, and F23 says the evidence base is absent rather than quoting it.^[27]
• Reversed: the claim this sensor was built to test"Practitioner sentiment is turning toward local-first" did not survive its own data. Posts arguing against the practice sit at 1–3% all window, so there was no opposition to turn from — the corpus is self-selected and structurally cannot answer the question asked. It answers a better one: who is doing the talking.^[4]
• Found in the collector: silent incompletenessThe first version of the new collector resumed per search term rather than per term-and-month. A run stopped part-way left a term looking finished, and the missing months would have read downstream as "nobody posted about this after November." Caught by checking window coverage rather than row counts, and fixed before any finding rested on it.^[4]

Data lives in a column-store database on a home server and is never pruned; history is the point of the instrument. Sources are fetched by purpose-built collectors that record every fetch, including the ones that failed, so a gap is visible as a gap. Full-text sources are stored as immutable originals with a hash, and the extracted text is regenerable from them.^[1]

Effort is counted across the whole project, not the drafting. It covers choosing the questions, standing up the collectors, the validation passes, the analyses that were killed, and the adversarial reviews — quoting only the writing sessions would describe a research program as though it were a blog post. Counts are programmatic from session transcripts, human prompts only, tool output and sub-agent turns excluded; transcript retention covers the instrument's whole life, so the count is complete rather than sampled.^[29]

What Revision 4 is not counted in, and why. The prompt tile above stops at Revision 3. Revision 4's session is deliberately excluded rather than estimated into it: reproducing the original count needs the same script against the same definition, and a figure produced two ways is worse than a figure with a stated boundary. Its model spend is missing from the dollar tiles for a harder reason — that work ran through a subscription harness rather than the metered gateway those figures come from, so no per-request cost exists for it at any price. The honest statement is that Revision 4's effort and spend are uninstrumented, not small. The instrument can measure what passes through its gateway and cannot measure what does not, and a provenance tab that quietly absorbed the gap would be claiming a completeness it does not have.

What it cost, precisely. The classifier ceiling runs, the model bake-off, the whole-corpus classification and the stratified reference sample: $5.52. The adversarial re-judge of 672 posts by a third model family: $1.90. The corporate-filings classification behind F2 and F22, run for the prior report and reused here: $10.61. Every figure is from the gateway's spend log, not estimated.^[29] Hardware is a home server already owned; its marginal cost is the electricity — which, per Finding 4, is the cheap and misleading number.

Platforms. Python collectors feeding a ClickHouse analytical database, queried in SQL. Drafting and both adversarial passes ran as Claude Code sessions against that database. The report is hand-authored MDX.

Models, by role. Claude Opus 5 and Claude Fable 5 (Anthropic) did research direction, drafting, and the red-team passes, under human instruction. Claude Opus 5 also served as the classification reference judge — measured against itself before use. deepseek-v4-flash (open-weight, via API) classified the 18,620-post corpus for $0.31, with its published levels replaced by the reference judge's where its error ran hot. gemini-3.5-flash re-judged the calibration sample during the adversarial pass, blind to the other two. No model generated any number cited as a finding without a measured error bar attached.

The prose of this report was written by an AI (Claude, Anthropic), working from the ledger and the sources, under human direction. The human owns the questions, the instrument, and every editorial judgment, and ordered both adversarial passes.

The numbers come from three places, and the report says which everywhere it matters. Most are deterministic database queries over a stored corpus. Some are quoted from named source documents with their own methodology. One finding's numbers (F3) are produced by a model classifying text — disclosed as method: the classifier was measured against a reference judge before use, its published levels are the reference model's, and an adversarial pass re-judged the same sample with a third model family and got the same answer. No model summarized evidence into a finding or graded its own work.

What that means for the reader: the framing and the sentences are machine-drafted and human-directed, while the quantities are auditable and re-runnable — and where a number could not be produced honestly, the report says it is missing instead of supplying one. An earlier revision of this disclosure denied the classifier existed; the red team caught it, and the correction is listed above rather than papered over.

• Rev 1 · Aug 15, 2026Organized around the instrument's own question ("is local-first cheaper?"), presented as a sequence of hypotheses checked. Staged privately, never published.
• Rev 2 · Aug 15, 2026Reframed to start from the business problem, with every term explained before use and one narrative through the evidence; the claim-by-claim accounting moved to Provenance. Two claims settled since Revision 1, both as mixed, and a third reframed from a sentiment turn — which its own data refuted — to a shift in who is talking. Staged privately, never published.
• Rev 3 · Aug 15, 2026 — the adversarial pass, and publicationA fresh red-team session attacked every finding before release. No finding died; five were reframed (F3's levels re-anchored to the reference judge, F7 recut around the same-toolbox comparison, F18's numbers replaced with a persisted query, F19's cost ground narrowed to the frontier tier, F20 scoped to offline experiments). Two provenance defects on this tab — an inflated source count and a false AI-disclosure sentence — were found, fixed, and documented above. Effort and cost figures were instrumented and added. This is the first published revision.
• Rev 4 · Aug 22, 2026 — the controlled study this report asked for, published by someone elseA bake-off designed to test this report's own headline was canceled when a sweep found the controlled version already in the literature. Three findings were added, all in Part C: a twelve-model supervisor benchmark over one fixed worker pool (F24), the permission-scoping failure that runs flat across every price tier (F25), and a large study where better delegation decisions produced no better work (F26). Two citations added. Two things were corrected rather than added. F7's central sentence had generalized from one model to all frontier models across three revisions, on a summary rather than the source's own results table — the numbers stand, the generalization is withdrawn, and the mechanism is published in its place. Position 1 in the Overview was sharpened accordingly: the controlled study finds the most expensive supervisor genuinely does win, so the position is now about the price of that premium rather than its existence. The source-count tile was restated from 65 to 68 on a stated definition, after the shelf beneath it was bulk-loaded with unrelated work. Neither correction came from a challenge; both came from re-reading sources already cited.
• Rev 5 · Aug 25, 2026A framing pass only. The report was retitled from "The orchestrator was supposed to be the expensive part" to the present title so the subject is named before the punchline, four evidence parts were re-headlined to state each part's takeaway rather than pose its question, the standfirst moved to the site's plain register, two residual first-person-plural phrasings were moved to the report's own register, and the index page now leads with the ambient claim this report measures. No finding, number, grade, confounder or citation changed — the evidence is identical to Revision 4.

A decision engine, plainly: the system that picks what to show, offer or approve for a specific customer in a specific moment — which options were eligible, which were withheld and why, what was recommended, in what order. It runs inside a page load or an ad auction, and it logs every decision.

One machine, many specialists. The instinct behind "ten agents" is ten deployments, and that instinct is what makes the idea sound unaffordable. It is not how the serving layer works. A single base model stays loaded, and each team's specialization is a small adapter file selected per request — measured at eight running in parallel on one base (F9). The frontier model stays in the picture as the exception rather than the default (F5, F7).

This is available as a hosted product as well as something to run on owned hardware — the adapter-serving mechanism is the same either way, which is what makes Step 1 below possible without buying anything (F6).

The loop that makes it compound. The previous diagram is a cost structure. It does not, by itself, improve. What turns a deployment into an asset is the harness around it: routing decided per step rather than per request, a mechanical check on whether the work functioned, and the verified results kept as training data for the next adapter.

Two things about this diagram carry the weight of the report. The gate is not a quality-assurance nicety — remove it and the loop does not merely weaken; training on unfiltered output measured worse than not training at all (F10). And the trace log is the compounding asset, not the model. It exists only if the harness belongs to the company, which is also why the shrinking exportability of frontier reasoning traces is a supply risk worth watching (F11, F12).

Step 1 · Take the free third before buying anything. Move routine, high-volume work to hosted open models and route the exceptions upward. This needs no hardware, no capital request and no data-center conversation. The measured saving is about a third at matched quality (F5), and the price gap between tiers is roughly twelvefold at the median (F6). A company that does only this step has captured the part of the argument that is real, bankable and reversible.

Step 2 · Find the gate before designing the loop. The single highest-leverage question in the whole program: for the work in question, what is the automatic test that says it functioned? Tests pass or fail. A ticket is resolved or reopened. A record validates or does not. Where such a test exists, a self-improving loop has a mechanism and the measured gains are large. Where it does not, the loop has nothing to filter on, and the evidence says training on unfiltered output makes things actively worse than doing nothing (F10). Choose the pilot domain by where the gate is strongest — not by where the enthusiasm is.

Step 3 · Run the pilot with one team, and instrument it before it starts. A workable first pilot is a single team with a repetitive, checkable workload, and four things captured from day one: a baseline taken before anything is deployed (productivity is unmeasured in the literature, F16 — the only credible number a company will ever have is its own, and it cannot be reconstructed after the fact); cost per completed task, not cost per million tokens; the gate's pass rate over time (F10); and the escalation rate to expensive models (F5). Record every step's decision and outcome, not just inputs and final answers — that record is the training data for everything that comes later (F11), and a system that did not log it has thrown the compounding away.

Step 4 · Price the supervisor on the pilot's own numbers — it is the cheapest experiment in the architecture. Once Steps 2 and 3 exist, this costs almost nothing: the gate already says whether the work functioned, the instrumentation already reports cost per completed task, and changing which model supervises is a configuration line. Run the same task mix with a cheaper model in charge and read the two numbers off the dashboard that is already running. The evidence says what to expect — measured over one fixed worker pool, running cost varied more than a hundredfold while management quality varied under four, and three expensive models were beaten outright by cheaper ones (F24). It also says not to expect the cheap seat to be free: one model collapsed entirely, so this is a test with a real failure mode and it should be run against the gate rather than against an impression. Do not assume the answer transfers from anyone else's benchmark — every published run of this kind is a single run on one task family, and the supervisor's job is specific to the work it supervises.

Step 5 · Scope permissions per task, from the first day. Grant an agent access to what the task in front of it needs, not to what its role might ever need (F15). This is far cheaper to build in at pilot scale than to retrofit across a company, and the standing over-privilege — not the model's behavior — is the exposure. Do not expect the supervisor to do this for you: across twelve models, granting the right file access was the one thing every one of them was bad at, and none cleared 50% (F25). Log what each worker was granted against what it actually opened — the gap is measurable from the first day and nothing else in the system reports it.

Step 6 · Only then consider owning hardware. The large cost multiples require roughly $45,000 of equipment kept genuinely busy, plus the engineering to run it (F4) — and the depreciation rate that would justify or sink that purchase is not yet measurable (F17). A company with a measured, high, steady volume from Steps 1–3 can evaluate this on its own numbers. A company without that evidence is buying a thesis.

Step 7 · Take this into a platform renewal as a narrower question. If an existing decisioning platform contract is up and the number causes sticker shock, the evidence here does not support treating agents as a replacement bid. Such a contract buys at least two things — an auditable, low-latency runtime, and the tooling to optimize it — and everything in this report bears on the second (F19, F23). The useful question at renewal is narrower and more answerable: how much of what is being paid for is optimization tooling, and does that half still earn its share now that agentic analysis costs cents? If the engine is not being actively optimized today, that half of the contract's value is already unrealized, and the comparison is against zero rather than against the vendor.

• Any promise of frontier quality from cheap models via smart routing. Twenty-one methods converge well below a perfect router and the gap does not close with effort (F13). Routing buys cost at acceptable quality, never parity.
• Any productivity multiple. Nothing in the evidence base supports one (F16).
• Any claim that the supervisor must be the most capable model. Measured over one fixed worker pool, the premium bought six points for four times the price, and three expensive models were beaten by cheaper ones (F24, F7).
• Any claim that a supervisor can be trusted to scope its own permissions. No model tested cleared 50% at granting the right file access, and the failure is flat across price tiers (F25).
• Any assumption that better delegation decisions produce better work. Doubling how often a supervisor picked the best worker moved finished quality by nothing measurable (F26).
• Any plan to teach the model company policies by training it on them. That is the configuration that produces fluent, confident, invented specifics (F14).
• Vendor-authored sufficiency arguments, including the one this report cites and grades as a position rather than a result (F8).
• Any architecture whose critical input is another company's optional output (F12).
• Any proposal to have agents make the decisions at request time. Latency and reproducibility each rule it out on their own; at volume, frontier-tier cost does too (F19).
• Any claim that building beats buying here, or the reverse. No outcome record has been published for this category (F23).
• Any plan to test changes on model-simulated customers. Use replay against real logged decisions instead (F21).

• Corrected · Rev 4F7's central sentence. Revisions 1–3 read the orchestration result as "prompting a frontier model to coordinate measurably fails." Re-mining the source at full text showed that claim rests on one model out of six tested in that seat — two others improved sharply under orchestration, and the outlier's collapse has a documented cause specific to it. The finding stands and its numbers are unchanged; the generalization has been withdrawn and the mechanism published in its place. Nobody challenged this. It was found by re-reading a source already cited, which is the uncomfortable part.
• Sharpened · Rev 4Position 1 in the Overview. It read "the supervisor does not need to be the expensive model," justified by a single vendor study. A controlled twelve-model benchmark now measures the same question, and its answer is more specific and less flattering to the original phrasing: the most expensive supervisor genuinely won. The position is now about the price of the premium rather than its existence, and it is justified by the controlled study rather than the vendor one.
• Added · Rev 4F24, F25 and F26, all in Part C, and two citations. They are numbered after the existing findings rather than inserted in sequence, because renumbering would break every cross-reference in the document; a reader meeting F24 immediately after F9 is meeting a later addition.
• Replaced · Rev 3F18's numbers. The original bars came from a drafting-session query whose term list was never persisted — a violation of this report's own provenance promise, caught by the red team. The finding survives with re-derived numbers whose query is stored on the ledger; the shape is unchanged, the levels moved.
• Reframed · Rev 3F3's published levels (they were the cheap classifier's, 6–10 points hot — now the reference judge's), F7's headline comparison (now the same-toolbox baseline), F19's cost ground (now scoped to the frontier tier), F20's scope (now offline experiments, explicitly). No finding was withdrawn; each reframe is described inside its finding.
• Removed · Rev 2The claim-by-claim structure of Revision 1. No finding was withdrawn; the accounting moved to Provenance, where the ledger belongs.
• Standing limit · the social corpus is self-selectedF3 measures who is talking, never whether anyone changed their mind. Collection capped bisection at four queries per month: 532 windows hit the cap and 22 remained censored at single-day granularity, which loses a window's oldest posts and therefore biases against finding an early rise. The red team verified the censored windows are balanced across years.
• Standing limit · volume trends are not usableTwo terms had their earliest months collected before the per-month cap was introduced, so those months are denser than later ones by collection method rather than by activity. Nothing in this report rests on a count of posts over time; F3 uses only ratios computed within a month, which that difference does not affect.
• Standing limit · keyword measurementsThe corpus trends (F1, F2, F18) rest on uncleaned keyword matching with no hand-labeled precision audit. Shapes are published; levels are not.
• Standing limit · single-source findingsF7, F9, F10, F24, F25 and F26 each rest on one published result. They are strong results with real controls; none has been independently replicated; and each finding names its domain, its authors' stake, and what does not travel beyond it. F24 is not a replication of F7 and should not be read as one — it measures models prompted to supervise, with no trained arm on its bench, so the two agree on the shape of the question and answer different halves of it.
• Standing limit · the trained-supervisor result is still unreplicated, and it is the load-bearing oneThe claim that coordination can be trained into a small model rests entirely on one vendor-authored study (F7) whose orchestrator was trained on the same tool catalog it was evaluated over, and whose one genuinely held-out-tool experiment saw every score collapse. The controlled twelve-model benchmark that arrived since (F24) does not test a trained supervisor at all. So the single most economically consequential claim in this report — that the coordinating role is cheap to buy because it can be trained rather than rented — has one source, one vendor, and no independent check. It is reported as strong because its internal controls are real, and flagged here because a buyer betting on it is betting on one paper.
• Untestable · the skill barrierWhether this pipeline is accessible to ordinary IT staff cannot be settled from literature — it is a claim about how hard work feels to a non-specialist. The adjacent evidence cuts against easy optimism: the measured adapter configuration needed compiler-level debugging (F9).

What this report is not claiming. Not that capital is flowing to this layer — only that disclosure language is. Not that hosted models beat owned hardware — the depreciation rate that would settle it is unmeasured. Not that small models are sufficient for most work — that remains a vendor's argued position. Not that any of this raises productivity, which nobody has shown. And not that the cost figures here are capability figures; price is not quality, and no benchmark in this report says otherwise.

1. DataThe claim ledger: 22 claims on the topic, each stored with its origin (external / the author's hypothesis / derived from reading), the method recorded with the claim, findings and conclusion kept as separate fields, refuted claims retained. The staging page renders its counts live from this ledger. Full-text reading shelf for this report: 68 sources (43 papers, 25 web documents) stored as immutable originals with content hashes, counted apart from the wider archive that shares the table. The stored location recorded against 1,244 of that archive's rows was found wrong during Revision 4 and repaired — see the audit note in How it was tested.
2. DataResearch corpus: 290,240 arXiv abstracts, January 2023 – August 2026, share-of-corpus routing measurement with the archive-growth null check (11,303 → 32,206 papers/quarter). Keyword uncleaned; shape published, level not.
3. DataCorporate filings corpus: 29,287 extracted passages, 2023-Q1 – 2026-Q3, balanced-panel restriction (companies filing in all four years), like-quarter comparison. Q1 2023 0.6% → Q1 2026 27.0%.
4. DataSocial corpus: 18,620 posts from 12,608 accounts, Sep 2025 – Aug 2026, 15 search terms (10 topic + 5 control), 1,081 queries, zero errors, resumable per term-and-month. Cap-censoring accounting: 532 windows capped, 22 censored at day granularity.
5. DataClassifier calibration: reference judge measured against itself first (98% on-topic / 97% speaker self-agreement, n=60, temperature 0); candidate bake-off scored against that ceiling; whole-corpus classification $0.31; monthly precision audit (67.5% → 75.5%, drift removed by conditioning on confirmed on-topic posts); published composition levels from the reference judge's 672-post stratified sample.
6. DataAdversarial re-analysis, second pass, Aug 15, 2026: a fresh session with no stake in the draft. Methods-level re-read of all 18 cited papers; every quoted number re-derived or traced to source; third-model re-judge of the 672-post calibration sample ($1.90, results within 1–3 points of the reference); censoring-by-month test; two provenance defects found on the trust page itself. Verdict file and re-judge artifacts stored with the analysis (REDTEAM.md, rejudge_x.py, rejudge_gemini.json).
7. DataInference price sensor: daily snapshots of a 413-model hosted catalog (prices, capability flags), begun mid-August 2026. Vendors overwrite yesterday's prices, so the series exists only because collection started before the question needed it. Cost projections in F19 are arithmetic over this catalog's published prices.
8. ExtPractitioner cost write-up (public forum, 2026): the $0.015/M-token marginal electricity figure with the author's own statement that it excludes hardware; ~$45,000 build cost from the public discussion beneath it. Graded: honest primary source, narrow scope.
9. ExtLLMRouterBench (arXiv:2601.07206): 400K+ instances, 21 datasets, 33 models. Up to 31.7% cost reduction at matched best-single-model performance; ~4% accuracy gain; named binary routers fail to trade cost for savings; pool-size diminishing returns. Graded: strong for the cost figure. Verified at methods level by the red team.
10. ExtToolOrchestra (arXiv:2511.21689, vendor research lab): 8B orchestrator on a Qwen3-8B base, GRPO-trained on outcome/efficiency/preference rewards, 16 H100s. 37.1% vs 21.2% (frontier model prompted to orchestrate the same expanded toolbox) and 35.1% (frontier model with basic tools only) on the text-only subset of a hard reasoning exam; +2.3/+2.5 points on two further benchmarks at ~30% cost. Same-backbone control: the untrained base model scores 30.6% in the identical full-pool configuration at 27.6¢ against the trained model's 9.2¢. Of six models tested in the orchestrating seat, only the one frontier model degrades under the larger toolbox; two others improve sharply (19.8→34.6, 14.0→32.8) — the paper attributes the outlier to vendor-family routing bias, measured at 98% of its delegations. Held-out-tool generalization collapses every score (its own 37.1→22.0) while widening the relative lead; the paper does not discuss the drop. ⚠️ The authors could not reproduce the frontier vendor's self-reported 84.2 on one benchmark, reaching 77.7 — at 84.2 their 80.2 would be a loss. Limitations exist only on the project page, not in the paper: unproven above 8B, untested outside reasoning tasks. Graded: strong result, vendor-authored, judge circularity noted (the frontier model also scores the outcome reward and sits in the pool as a tool); same research group as c11 — counted as one voice with it. Re-mined at full text for Revision 4.
11. ExtSmall-model sufficiency position paper (arXiv:2506.02153, vendor research, v2 Sep 2025): explicitly a position, not a benchmark. Graded: contested; used only as framing, never as evidence of sufficiency. Same research group as c10.
12. ExtMulti-adapter serving report (inference project with a cloud provider, Feb 2026): GPT-OSS 20B base, 8 adapters in parallel, LoRA rank 32, 1,600 input / 600 output tokens, vLLM 0.15.0 — 144 OTPS / 135 ms TTFT, 171 / 124 tuned; the 10× TTFT compiler-recompilation regression and its one-hint fix. Graded: strong for the configuration, vendor-published. All numbers verified verbatim by the red team.
13. ExtAdapter-store serving documentation (a second inference stack): the same per-request adapter selection mechanism, documented independently of c12.
14. Ext"The Verifier is the Curriculum" (arXiv:2607.09709, academic — Science Tokyo / Zhejiang / NUS): execution-gated self-distillation on video-game project generation, Qwen3-14B + LoRA. 8.8% → 42.2% over three gated rounds (all significant); gold-duplication control regresses to 5.6% (p=0.019); gate-swap control (lenient check passing 99.9%) erases the entire gain (p=1e-3); coverage 18/25 → 25/25. Graded: strong; the decisive control is what earns it; single model, single benchmark, N=25 — the design rule travels, the effect size does not.
15. ExtHarness-native routing architecture (arXiv:2607.11399, open-sourced Jul 2026): architectural argument, not a benchmark. Graded: moderate; used for framing.
16. ExtPublic issue tracker of a frontier provider's tooling, counted Aug 15, 2026: 18 open issues on reasoning summaries, one documenting encrypted-by-design reasoning content. Graded: observational; establishes the pattern, not the cause; count not independently re-verified.
17. Ext"The Routing Plateau" (arXiv:2606.07587, university + industry): 21 methods, 5 benchmarks, one unified setup. Best router trails oracle by 10–30 points; top-15 spread 0.23 points on one benchmark; kNN top-2 on all five; plateau persists cost-aware; authors' remedy closes 14.6% of the gap. Graded: strong; the central negative result of this report. Verified at methods level.
18. ExtFine-tuning and hallucination mechanism study (arXiv:2604.15574 — interference among overlapping representations; freezing mitigation, which suppresses new-fact learning and is scoped by its authors to settings where that is acceptable); controlled tuning-vs-retrieval comparison (arXiv:2604.23801 — +6.8 points at 4B scale, retrieval null); LoRA configuration ablation (arXiv:2605.28222). Graded: mixed — they disagree, and the disagreement is reported rather than resolved.
19. ExtDynamic capability scoping (arXiv:2607.22445, independent researcher, workshop): 600-prompt dataset over a 15-permission taxonomy, κ = 0.917/0.967 on a reviewed sample; ceiling violations 46 → 3 by iterating dataset against policy. Graded: strong prior art on the framing; the architecture is proposed, not implemented.
20. DataDecision-support framing share: re-runnable corpus query stored on the ledger (claim decision-support-framing-share) with its full term lists. 2023: 0.68% of all papers / 10.3% of agent papers; 2026: 1.46% / 7.5%. Replaces Revision 2's unpersisted drafting-session numbers.
21. ExtLong-history user transformers for real-time ad ranking (arXiv:2607.14331, production advertising system): the few-hundred-millisecond constraint quoted verbatim; offline encoder + cached representation + lightweight runtime model; 72–80% offline quality retention of an unservable full-history model; deployed via live A/B. Graded: strong, production. Shares an organization and an author with c23 — counted as one production culture with it.
22. ExtVariance-components decomposition of LLM non-determinism (arXiv:2607.13304): 12,933 responses, fully crossed; four sources — resampling, prompt phrasing, model identity, language. Graded: moderate for this use — the task is brand-sentiment measurement, so it is cited for the structure of the variance, not the magnitudes.
23. ExtGryphon-v2 generate-and-rank recommender (arXiv:2608.06213): replaces a 15-generator production cascade; distills a training-only teacher into the serving module explicitly to avoid a second model on the serving path; deployed via online A/B. Graded: strong, production. Shares an organization and an author with c21.
24. ExtRecHarness bandit-routed agentic harness (arXiv:2607.29241): Thompson-style routing over offline experiment directions; ablation vs model-chosen directions (improving-trial hit rate 47.9% vs 21.7%); one live element tested a single already-chosen candidate. Graded: strong for the offline split; not evidence about live-traffic exploration, and cited accordingly.
25. ExtOff-policy evaluation literature (964 papers, 2023–2026, from the research corpus); logging-policy design and the reward–coverage tradeoff (arXiv:2605.15108); replay-to-launch-readiness framework (arXiv:2605.12840 — offline improvement is evidence for a live test, not a substitute; benchmark-log study). Graded: mature method literature.
26. ExtFour studies refuting LLM-simulated survey respondents: cross-domain benchmark vs non-LLM baselines (arXiv:2607.26348 — no model beats the strongest baseline at the individual level; its own discussion names offer personalization as the decision at risk); distribution-first population simulation (arXiv:2607.18310 — 85% collapse on 2,414 respondents; a measured mitigation exists but over-disperses; includes a small booking-task choice probe showing the same collapse); urban-publics behavioral replication (arXiv:2607.27100 — 1 of 8 models met the authors' prespecified criterion against 843 respondents); belief-update simulation (arXiv:2607.28347 — all 6 models fail from self-generated starting points; 4 of 6 pass given true starting points, so the failure is the cold-start regime this use case requires). All four verified at methods level by the red team.
27. DataPrior adversarial re-analysis (Aug 11, 2026, of the preceding report): seller-cohort and false-positive audit of the 16-company decisioning panel; placebo persistence panel (mainframe 100%, on-premise 91.7%, decisioning 87.5%, blockchain 77.3%, digital transformation 72.0%, metaverse 42.9%); exclusion of a widely quoted vendor-outcome statistic as inadmissible. This report inherits those kills; the second red team verified the inherited rewrite reproduces the prior verdict exactly and annotated the one place on the ledger where the dead reading had been reused without it.
28. ExtBuild-vs-buy decision framework (arXiv:2606.29816): cited from its abstract, where the quoted cold-start phrase appears verbatim; not on this report's full-text shelf. Graded: honest framework, concedes the missing evidence base.
29. DataEffort and cost accounting: human prompts counted programmatically from Claude Code session transcripts (tool results and sub-agent turns excluded), 425 prompts across 30 sessions, Aug 3–15, 2026, covering the project through Revision 3; transcript retention covers the instrument's entire life, so that count is complete rather than sampled. Revision 4's session is excluded from it rather than estimated in, and Revision 4's model spend is absent from the dollar figures below because that work ran through a subscription harness rather than the metered gateway — uninstrumented, and stated as such rather than absorbed. Model spend from the LiteLLM gateway's per-request log: $5.52 report evidence work, $1.90 adversarial re-judge, $10.61 prior-report filings classification, $19.10 instrument lifetime.
30. ExtClawArena-Team (arXiv:2606.31174v2, July 2, 2026, academic — UNC-Chapel Hill / UC Berkeley / UC Santa Cruz): twelve models placed in the supervisor seat over one fixed, locally-served worker pool. 41 multimodal multi-directory scenarios, 258 scored rounds, 72 staged mid-run updates; the main agent is constrained to text-only perception and partial workspace access so delegation is forced. Management score = task completion × the mean of four least-privilege and routing components. All scoring is execution-based with declared exit codes — no model grades any outcome. Cost $0.80–$93 per run (>100×) against a score span under 4×; workspace-permission precision never reaches 50% for any model (best 49.2); ten models cluster in a 9.9-point band while per-subagent forbidden-access rates spread roughly an order of magnitude. Graded: strong — the controlled version of F7's question, and the only source here that isolates the supervisor with an execution-checked score. Four author-stated limits carried into F24: the top score is a model-plus-refusal-fallback composite, self-hosted costs are imputed from a public rate card and called approximate, every model was run once with no variance estimate, and the fixed pool ties all results to one level of worker capability.
31. ExtDecisionBench (arXiv:2605.19099v1, May 18, 2026, academic — Penn / Columbia / MIT / Stanford / OpenMesh AI): fixed task suite (GAIA, τ-bench, BFCL v4 multi-turn), fixed 11-model / 7-vendor peer pool at a pinned price date, fixed delegation interface; only the supervisor's information about its workers varies across five conditions. n = 23,375 task instances. Best-worker selection rate 14.2% (blind) → 29.5% (on-demand profiles); end-task quality flat, mixed-effects |β| ≤ 0.010 with all p ≥ 0.21 on the full sample. Perfect-delegation ceiling sits 15–31 points above every observed configuration and stays positive under a 70%-realization stress test. Vendor self-preference 1.5–3.7× chance for several models; one major vendor neutral. ⚠️ Graded: moderate, and the grade is about the denominators — the 29.5% rests on 122 delegation decisions, awareness reduces delegation frequency, one information format scores 7.5% (below blind), and no confidence interval is published on those rates. The flat-quality null is the robust half and is what F26 rests on. Authors' own limits include single-seed evaluation and near-zero delegation on one of the three suites.