Letting AI agents tune the system: adopted fast, measured almost never

The proposal goes like this. A decision engine — the rules-and-models system that picks which offer to show, which price to quote, whether to approve an application — stays exactly as it is: deterministic, inspectable, with no AI agent called while a customer is waiting. Every decision it makes is logged: what the options were, which one was chosen, what was suppressed and why, and what the customer did next. Then, entirely offline, a team of AI agents reads those logs. They look for patterns, run simulations, propose experiments, and suggest changes to the engine's configuration. A person approves or rejects each proposal before anything ships.

The boundary is the interesting part. Putting a language model inside the decision path destroys the property that makes a decision engine defensible: you can no longer reproduce why a particular customer got a particular answer. This proposal keeps that property and puts the intelligence somewhere it cannot contaminate the record. That is a genuinely thoughtful piece of design, and it is why the strategy deserves a real examination rather than a dismissal.

The examination is worth doing because the strategy is not hypothetical. In the filings of a 529-company panel of large public corporations, the number describing AI agents alongside decision-making or optimization language went from three companies in the first eight months of 2024 to 107 in the same window of 2026 ^[3]. That is one company in five, describing this to investors under legal liability. Something is being adopted at speed, and the question of what it buys is worth an evidence base rather than a brochure.

So this report asks a narrow question with an answerable shape: across the published research, what happens when an organization does this — and how would anyone know?

Three evidence bases, all public or self-collected, all queryable after the fact.

The research base is an abstract-level corpus of arXiv — a public preprint archive where most machine-learning research appears first — covering January 2023 through August 2026 ^[1]. Abstracts alone cannot answer a question about method, so the corpus is used only to locate papers; every paper this report rests on was fetched as its original document, converted to text, and stored with a content hash ^[2].

The adoption base is a corpus of filings to the U.S. Securities and Exchange Commission, the federal regulator that public companies report to. They are drawn from EDGAR — the Electronic Data Gathering, Analysis, and Retrieval system, the free public archive where every one of those filings is published ^[3]. Filings are split into sections, so what a company says it is building can be counted separately from what it says it is afraid of.

Each paper was read by a language model against a fixed extraction schedule, and the model's job was constrained to reporting what the paper says rather than judging it. It had to name what was measured, against what baseline, how outcomes were scored, whether the headline claim was stated before the data was seen, and — the field that did the most work — which measurement approach the authors report as having failed.

• Tested · the reading model, against an answer key written in advance

  Ten papers were read end to end in a supervised session, and an answer key written from those reads before the seven candidates were run — covering only checkable things: whether the paper tests its own headline claim, who or what it studied, how far it transfers, and whether it reports evidence on productivity at all. Seven models then read the same ten. The winner scored 46 of 50. The decisive column was not accuracy but invention: only two models never claimed a productivity finding in a paper that contains none ^[4].

• Failed · one model disqualified for fabricating supporting evidence

  A cheap, otherwise respectable model scored 36 of 50 on the factual audit while manufacturing productivity evidence in six of the eight papers that have none. For a corpus sweep whose entire purpose is to find out whether a claim is supported, that is disqualifying, and it would have been invisible without the answer key. A premium model at three times the winner's measured cost scored lower than the winner ^[4].

• Failed · a formatting artifact that first read as two models being incompetent

  Two models initially scored zero and 36 because they wrapped their output differently from the rest — one nesting its answers under headings, one fencing its output as a code block. Both were scoring bugs in the harness, not model failures, and both were found only because a zero was too extreme to believe. The scores reported here are after the fix.

• Null check · does the corpus grow just because the archive grows?

  Research volume roughly doubled over the window, so a rising count of relevant papers proves nothing. Measured as a share of all papers instead — using the materialized queue as the numerator, so the figure is re-runnable — the organizational seam moves from 0.31% in the second half of 2024 to 0.48% in 2026 ^[1] — a real rise, and a modest one. The honest reading is that attention is increasing slightly, not that the field has turned to face this question.

• Null check · does the filings growth come from more companies filing?

  The adoption numbers in Finding 1 would be an artifact if the panel grew. It did not meaningfully: 511, 520, 525 and 529 companies filed in the January-to-August window of 2023 through 2026 respectively, while companies naming agentic AI went 6, then 55, then 125 ^[3]. All comparisons are like-for-like eight-month windows for the same reason.

• Tested · the denominator was written down before the reading started

  The list of 1,259 papers was materialized to a table before the sweep began, so progress and coverage are measured rather than estimated, and the sweep can report honestly that it finished. Nine papers failed permanently: their identifiers exist in the abstract corpus but the archive no longer serves the documents. They are counted as unread rather than quietly dropped.

• Failed · the first sweep lost 188 papers to rate limits and reported success

  The initial pass had no backoff, logged transient service errors as failures, and finished having read 1,061 of 1,259. Because it exited cleanly, nothing about its completion message indicated a problem. The gap was visible only by comparing the tagged count against the stored denominator — which is the argument for storing the denominator.

• Tested · three papers re-read independently where the claim is load-bearing

  The two production offline-evaluation papers behind Finding 13 and Finding 14, and the meta-analysis behind Finding 4, were re-read in full rather than accepted from the extraction, because the report's most actionable positions rest on them. This was a second AI pass under human direction, not a human reading; see the AI disclosure. All three survived; two of them gained detail that strengthened the finding.

Effort counts are programmatic, taken from session transcripts, excluding tool results and skill invocations. They cover this report's research and its adversarial pass — deciding the question, the model bake-off, the corpus sweep, the second extraction pass, the drafting, and the Revision 2 re-derivation: 14 prompts in the research session, 3 in the adversarial one. They do not include building the instrument this runs on: the research corpus, the filings corpus, the storage layer and the collectors were built earlier and are shared with previous reports. Transcript retention covers this report's window completely.

Spend breaks down as $1.24 for the seven-model bake-off, $13.66 for the corpus sweep including the papers the first pass lost to rate limits, $0.27 for the measurement-method pass over the shortlist, and $0.10 for the adversarial pass's 60-paper cross-model replication. Models by role: one hosted model read and extracted from all 1,250 papers; six others were used only in the bake-off that selected it; the bake-off's runner-up, from a different model family, re-read the replication sample; the prose was drafted by a language model under human direction. Reading ran on a dedicated machine, capped so it could not compete with anything else, and wrote a progress record per paper so that a stall could be told from a finish.

The prose of this report was written by an AI system under human direction. The question, the scope, the framing and every editorial decision are the author's; the sentences were drafted by a model and revised by a human.

The numbers were not produced by an AI. Counts come from deterministic queries against stored corpora, re-runnable and version-controlled. Where a model sits in the pipeline — reading full text into a fixed schedule of fields — it was selected by a bake-off against an answer key written in advance, and the one model that fabricated supporting evidence was disqualified before the sweep began. Every figure quoted inside a finding was traced back to the source paper by a second, independent read.

No human read the source papers, and this report does not claim one did. The author set the question, chose the scope, directed every pass and reviewed the output; the reading was done by AI throughout — including the adversarial re-derivation behind Revision 2, which was a separate AI session with no stake in the draft, working from the stored corpora and the papers' full text. Two consequences a skeptical reader should weigh. The answer key that selected the reading model was itself produced by an AI, so it measures agreement with a careful second reader rather than with ground truth — and one of the seven candidates shares a model family with the reader that wrote it. And the "independent re-read" that checks the load-bearing figures is independent of the bulk extraction pass, not of AI judgment in general.

What that means for the reader: the argument is auditable in the ordinary way. Every number carries a citation naming the dataset or the paper it comes from, the failed tests are listed beside the passed ones, and the extraction schedule is published so the same corpus can be re-read and disagreed with.

Revision 3 — Aug 25, 2026. A framing pass only. The report was retitled from "The optimization loop" to the present title so the strategy is named before the verdict, three evidence parts were re-headlined to state each part's takeaway rather than name its topic, the standfirst moved to the site's plain register, and the index page now leads with the ambient claim this report measures. No finding, number, grade, confounder or citation changed — the evidence is identical to Revision 2.

Revision 2 — Aug 19, 2026. Adversarial re-derivation by a separate session with no stake in the draft; every self-collected number re-queried independently, all 20 external citations re-derived from the papers' full text, and a 60-paper sample re-extracted through a second model family. No finding fell. What changed: Finding 9's consistency figure corrected from "one in 22 — 7.1%" to the paper's actual 1 of 22 (4.5%); Finding 13's wasted-experimentation figure corrected from 14 to more than 20 weeks a year; Finding 10 relabeled — its confidence and accuracy numbers come from a non-randomized within-subject study, its recovery numbers from a separate randomized one; Finding 8's survey described as open-source-community self-report rather than an enterprise field study; the r = −0.45 correlation attributed to the single industry study the systematic review relays; Finding 11's "exceeded the humans' own" scoped to the study that had a human-alone arm; Finding 12's comprehension gap labeled as the widest of seven measures; Finding 15's parallel-voting recommendation now carries the same paper's saturation warning. Provenance corrections: the corpus null check restated from the stored queue (0.31% → 0.48%), the bake-off price comparison corrected from ten times to three times measured cost, the effort and spend tiles updated to include the adversarial pass.

Revision 1 — Aug 19, 2026. First publication. Three papers were removed from evidence during drafting and retained as examples in Removed & limits.

1. Capture a baseline before anything is switched on. It cannot be reconstructed afterward, and the studies that lacked one could not separate tool effects from the fact that early adopters were already more active (Finding 6).
2. Disqualify self-report as a primary outcome. Keep it as a secondary measure and treat its gap from the observed number as data (5, 6, 9).
3. Measure in the real environment, not a controlled pilot. The effect gradient from laboratory to production is the distance between 0.73 and 0.01 (4).
4. Refuse volume metrics as evidence. Their error changes sign between settings (7).
5. Time the whole loop, from proposal to shipped or rejected. Work created downstream of a proposal is invisible to a step-level measurement (8).
6. Track agreement and catch rate separately, against planted errors. A single approval rate cannot distinguish a good reviewer from a compliant one (10, 11).
7. Screen proposals offline with importance-sampling estimators, and validate the estimator against live outcomes before trusting it. The reward-model implementation is the one that fails (13).
8. Keep live testing as confirmation, never replace it. False negatives cost more than false positives (13, 14).
9. Run periodic unassisted tasks to detect capability decay in the reviewers. Nothing else detects it (12).

One — separate the two halves of this strategy, because they are at completely different stages of maturity. The offline-evaluation component is validated in production by at least two published systems with correlation figures against live outcomes (Finding 13, 14). The agent-layer component is supported by benchmark studies whose magnitudes do not transfer (15). An organization can adopt the first without the second, and the first is where the measured return is.

Two — treat the productivity claim as unestablished, in either direction. It is not refuted; it is unmeasured in working environments (3, 4). Anyone quoting an organizational productivity figure for this strategy is quoting a laboratory, a vendor, or a survey — and the survey is demonstrably the least reliable of the three (6).

Three — build the measurement before the loop. The corpus offers three times as much guidance on measurement as on outcomes (3), the single irrecoverable step is the baseline, and every failure mode documented here is invisible to the instruments a team would reach for first (7, 8).

Four — instrument the humans, not just the agents. Three of the four measurement points the evidence supports sit on the human half of the loop, which is the half that is almost never measured (10, 11, 12).

Three papers were removed from evidence during drafting. All three were initially attractive, all three appear in earlier drafts, and each is retained here as an example rather than a citation.

• Removed · Rev 1 · scaling claim never tested

  A paper proposing an "inverse-wisdom law" — that adding more checking agents increases the stability of wrong answers — was dropped as evidence. Its abstract advertises a scaling result across 12,804 recorded runs. Every experiment uses exactly three agents; the number never varies, and the authors concede in their limitations that validation beyond three agents remains future work. Its three headline metrics also turn out to be one measurement partitioned in two: two of them sum exactly to the third, row after row. Kept as the clearest example in this corpus of an abstract that names a law the paper does not test.

• Removed · Rev 1 · construct not what the abstract implies

  A study of hidden coordinators in agent teams was reduced from evidence to a caution. Its headline effect is a keyword composite whose formula subtracts counts of sexual-content words, computed on a scenario involving coerced disclosure and atrocity justification rather than anything resembling office work. Its central result is conceded to be post hoc rather than predicted, and the striking claim that internal distortion was invisible to output-based evaluation is, in the authors' own limitations, a test that could not be run because every condition scored at ceiling.

• Removed · Rev 1 · population mismatch

  A bibliometric study of 141,550 scientific papers, finding an inverted-U between the share of a team credited with conceiving the work and the team's citation impact, was dropped. It is a well-executed study of authorship credit in natural science between 2007 and 2015, and the distance from there to agent teams optimizing a decision engine is too great to bridge honestly.

• Tested · Rev 2 · adversarial re-derivation, no finding fell

  After first publication, a separate session with no stake in the draft — an AI session, like every pass in this report — re-derived every number here. It re-queried the self-collected series with independently written searches (an independent term list moves the 107-company count to 119 and the panel checks reproduce to the digit), re-derived all 20 external citations against the papers' full text, and re-extracted a 60-paper sample through a second model family aimed at Finding 3 and 15, the two findings Revision 1 named as its weakest. No finding fell. Two figures, one design label, one setting description and a handful of provenance sentences were corrected — each one listed in Versions. The two hard numeric errors it caught had survived the second read, which is the strongest argument this report can offer for adversarial passes.

• Limit · the corpus window

  The abstract corpus begins in January 2023, so foundational earlier work is absent by construction. Anything published before that window is outside this report's reach unless it was cited into it.

• Limit · nine papers unread

  Nine of the 1,259 papers in the queue could not be retrieved: their identifiers exist in the abstract corpus but the archive no longer serves the documents. They are counted as unread rather than dropped from the denominator.

• Limit · the search is keyword-based

  Papers were located by matching organizational and coordination language against titles and abstracts. Relevant work that avoids that vocabulary is missed, and the seam deliberately excludes a second, larger body of multi-agent coordination research that was scoped but not read for this revision. The 84% non-transfer rate is a property of this search, not of the field.

• Limit · single-reader extraction, now sampled by a second reader

  Each paper was read once, by one model, selected by bake-off. In Revision 2 a 60-paper sample was re-extracted through a second model family: the headline non-transfer rate reproduced (81% against 87%), the evidence ranking's top and bottom held, per-paper labels agreed 79% of the time, and the middle counts of Finding 3's chart moved by up to a factor of two. The full-corpus counts remain one model's reading, now with a measured error band — and every pass in this report, including the checking passes, was performed by an AI.

1. DataResearch corpus: 290,240 abstracts from arXiv, a public preprint archive, January 2023 – August 2026. Share-of-corpus normalization used for all trend claims, with the archive-growth null check (0.31% in H2 2024 → 0.48% in 2026, queue-count over corpus-count, re-runnable). Keyword search uncleaned.
2. DataFull-text reading sweep: a 1,259-paper queue materialized to a table before reading began; 1,250 read at full length as original documents with content hashes, nine permanently unavailable. Extraction schedule published with the analysis; per-paper progress written as durable state. Distribution: 1,057 no transfer, 193 partial transfer, 142 transferring with a headline claim judged to hold; verdicts 73% hold, 16% overclaimed, 10% untested. Rev 2: a 60-paper sample re-extracted through a second model family reproduced the non-transfer rate and the ranking's extremes; per-paper label agreement 79%.
3. DataCorporate filings corpus: filings to the U.S. Securities and Exchange Commission via EDGAR (Electronic Data Gathering, Analysis, and Retrieval), its public filing archive, split into sections; 529-company panel, 4,364 sections in the January–August 2026 window. Like-for-like eight-month windows throughout. Panel-stability check: 511 / 520 / 525 / 529 companies filing in the same window, 2023–2026.
4. DataModel bake-off: seven models over the ten papers read end to end in the supervised session, 70 calls, $1.24, scored against an answer key written before any model output was seen. Winner 46/50 with zero invented findings; one model scored 36/50 while fabricating productivity evidence in six of the eight papers containing none; a premium model at roughly ten times the price scored 38/50. Two initial zero-scores traced to harness formatting bugs and corrected.
5. ExtMeta-analysis of generative AI effects on developer productivity and learning (arXiv:2605.04779): 23 studies, 27 effect sizes; 14 studies with 3,535 participants and 6,355 repositories for productivity. Pooled g = 0.33 [0.09, 0.58]; laboratory g = 0.73 (p < 0.001); enterprise g = 0.19 (p = 0.448); open source g = 0.01 (p = 0.975). Graded: strong; the central quantitative result of this report. Re-read in full, independently of the extraction pass.
6. ExtRandomized controlled trial of early-2025 AI tooling on experienced open-source developers (arXiv:2507.09089), conducted by a nonprofit AI evaluation organization: 16 developers, 246 issues, repositories averaging over a million lines and ten years of history. 19% slowdown; forecast 24% speedup, post-hoc estimate 20% speedup; economists 39%, machine-learning experts 38%; 75% of developers slowed; 47% more lines of code per forecast hour; under 44% of generations accepted; 9% of time reviewing generated output. Graded: strong; the only randomized field trial in the corpus. Authors scope the result to experts in mature codebases.
7. ExtTwo-year longitudinal study of developer productivity with and without an AI coding assistant (arXiv:2509.20353): 105 weeks of commit activity, September 2022 – September 2024, paired with perception. All users perceived no change or improvement; commit activity showed no significant change (rho = 0.17, p = 0.406). Graded: moderate; before-after design with self-selected adopters, a limitation the authors name.
8. ExtPractitioner survey on productivity with an AI assistant (arXiv:2602.03593): 86% satisfied or very satisfied, ~60% reporting under one hour saved per week, correlation r = 0.34. Graded: moderate; self-report only, cross-sectional.
9. ExtMulti-case study of generative AI in agile teams (arXiv:2602.13766): 13 months, October 2023 – November 2024; completed story points 281 → 447 (+59.1%); committed lines of code flat (p = 0.928); code-quality metrics moving in opposite directions across squads. Graded: moderate; before-after, few cases, but the divergence is the point. Authors state that activity measurement alone would have concluded no impact.
10. ExtQuasi-experimental study of generative AI in collaborative open-source development (arXiv:2410.02091): 24 months, January 2021 – December 2022; matched design; increased merged contributions alongside an 8% increase in coordination time; raw defect counts rising while per-contribution defects stayed flat. Graded: strong for the coordination-cost result.
11. ExtSystematic review of large language model assistant effects on developer productivity (arXiv:2507.03156): 39 primary studies, literature window January 2014 – December 2024, mapped to a five-dimension productivity framework. Suggestion-acceptance rate misleads in isolation; relays a 70-company industry study reporting throughput correlating negatively with code quality at r = −0.45. Graded: strong for the metric critique; a review, so it inherits its inputs' limitations.
12. ExtSurvey of developer productivity with generative AI across five dimensions (arXiv:2510.24265): two-week survey, 415 developers across 56 open-source communities; effort redistributed into review burden and verification load behind an appearance of speed. Graded: moderate; self-report, short window.
13. ExtLongitudinal mixed-methods study of AI coding assistants (arXiv:2605.23135): two questionnaires six months apart, 158 and 101 respondents, matched cohort of 95. 84% reported improved productivity at both points; the share reporting a worse experience in at least one dimension rose 14% → 27%, with no recovery out of that group; 82% reported writing less code. Names "supervisory engineering work" as an emerging category. Graded: moderate; self-report, and the authors state the 84% describes continuing users only, since anyone who abandoned the tools never entered the sample.
14. ExtStudy of human grading consistency (arXiv:2409.12967): 28 graders, two batches of 20 assignments, with a duplicate planted so each grader re-graded the same work. 24 of 28 (85.7%) self-reported consistency; of the 22 who never noticed the duplicate, 1 (4.5%) gave identical grades; one further grader matched grades only after spotting the repeat. Standard inter-rater agreement statistics are inapplicable at two ratings per item, and self-review did not improve consistency. Graded: strong; the design is what earns it.
15. ExtTwo-part study of large language model explanations in human decision-making (arXiv:2604.03237): a within-subject study (n = 27) with objective accuracy 49.8% → 48.8% between prediction and explanation stages while confidence rose from a 3.66 baseline to 3.81; a separate randomized study (n = 100) with error recovery 16.2% in the explanation condition (condition differences p = .055); a selective-automation rule using a prespecified ten-point probability gap reached 69.5% in post-hoc evaluation. Graded: strong for the confidence-accuracy divergence; design labels per study, as corrected in Revision 2.
16. ExtClinical case study of expert reliance on AI assistance (arXiv:2502.03482): radiologists rated the system 2.4/5 and 2.1/5 across two studies while its measured accuracy (69.3%, 76.0%) exceeded their unaided 63.2% in the study that measured both (the second had no human-alone arm); performance feedback before decisions did not significantly improve combined performance; in the independent-judgment-first workflow radiologists kept their initial opinion 79.6% of the time when the machine disagreed, with feedback varied concurrently. Graded: moderate — medical imaging, so the mechanism transfers and the magnitudes do not.
17. ExtEmpirical study of user trust and team performance in plan-then-execute agent systems (arXiv:2502.01390): stated trust 3.52/5 against calibrated trust of 0.03–0.27 on high-risk tasks with imperfect plans; strict action-sequence matching reported as a metric that broke by penalizing benign redundant actions. Graded: moderate; simulated tasks.
18. ExtRandomized study of coding agents against chatbot assistance (arXiv:2607.26375): initial accuracy effect size d = 1.4 favoring the agent; comprehension deficit d = 0.9 overall, widest on code-specific questions at 0.642 against 0.951; self-rated understanding 3.3 against 4.2; agent rated more helpful at 4.7; on an extension task without the agent (chatbot available to both groups) the advantage reverses when initial differences are controlled. Graded: strong; laboratory, ~2 hours per participant.
19. ExtStudy of expert performance in self-organizing multi-agent teams (arXiv:2602.01011): ranking-error and benchmark-accuracy measurements showing self-organizing discussion failing to leverage expertise, with the correct baseline established as the single best individual rather than the group average. Graded: moderate; benchmark tasks.
20. ExtOff-policy evaluation in production at Adyen, a payment processing company (arXiv:2501.10470): a year of completed live A/B tests compared against offline estimates over billions of logged interactions. Inverse propensity scoring and its self-normalized variant sustained Pearson correlation above 0.8; the direct reward-model method correlated negatively; the doubly robust estimator was dragged to near zero by inheriting that reward model; variance largely disappears above one million records. 58% of the company's A/B tests were flat or inconclusive, quantified as more than 20 weeks per year. Estimated 9–54 million incremental transactions over six months from faster identification of winners. Graded: strong, production, self-reported by the operator. Re-read in full, independently of the extraction pass.
21. ExtUser simulation for evaluating preference-elicitation policies at a large recommender platform (arXiv:2409.17436): simulation predicted +1.36% [−0.11%, 2.83%] selections against +1.17% [−0.03%, 2.36%] measured post-launch across 500,000 users; three sub-period live experiments each ordered three candidate policies differently, only one matching the full launch, while the simulation matched it. Authors name counterfactual robustness of the user model as the precondition. Graded: strong, production. Re-read in full, independently of the extraction pass.
22. ExtControlled study of interaction topology in multi-agent decision systems (arXiv:2605.01147): 5,760 synthetic credit applications, four fixed roles, 24 orderings, model families from 3 billion to 72 billion parameters. Approval rates spanning 59 percentage points on the smallest model and 21 points at 70 billion and above; agreement above 99.9% with error correction below 0.1% at the largest scale; parallel independent voting eliminating the ordering effect while saturating at 98% approval on the smallest model. Graded: moderate — a position paper, single synthetic domain, internally inconsistent in places; the weakest of the three agent studies, and Finding 15's direction survives without it.
23. ExtEmpirical study of diversity in multi-agent idea generation (arXiv:2604.18005): 1,000 proposals per configuration across 20 topics; junior-led horizontal structures maximizing measured diversity, designated-senior and interdisciplinary structures minimizing it; per-agent diversity utilization falling from 1.03 to 0.47 as groups grew from three to seven. Diversity metrics validated against five human annotators at 87% agreement for the primary metric; idea quality scored by a model judge, not by humans. Graded: moderate — the quality half is unvalidated by humans, and the authors say so.
24. ExtSystematic study of scaling behavior in multi-agent systems (arXiv:2606.00655): agent counts one to eight, two model families, with a token-padding control isolating coordination overhead from context length. Inverted-U performance curve peaking around four to five agents; quadratic token cost. Graded: moderate — 17 questions per subject with three repeats, small enough that reported differences of a few points represent a handful of items.