Papers to end all papers
What six months of preprints on large language models and AI agents in condensed matter, materials and adjacent quantum research look like, measured with AI referees. Companion data to a commentary in the Journal Club for Condensed Matter Physics.
Pace
What the papers are about
All 304 core papers were coded from title and abstract by GPT-6.1 Sol and Claude Opus 5.5 coders working independently (κ = 0.95 for the field, the research community whose expertise is needed to judge the physics, and 0.92 for the question). A paper restricted to one class of materials belongs to that class, otherwise to the community of the technique it concerns. Effective number of fields 13.5, of questions 5.4. Abstracts understate physical content: papers whose AI use appears only in the body are not found by the searches at all.
Model obsolescence
Median lag from the release of the newest model a paper ran to its first arXiv version: 96 days (IQR 47–213; n = 189), with a median of 6 flagship releases in between. 4% used the current flagship; 17% ran a newest model more than a year old.
See panel (b) of the pace figure.
What goes wrong
One GPT-6.1 Sol referee per paper read the full text of 228 arXiv papers. The 454 major issues they raised were coded into a taxonomy (κ = 0.94 between Sol coders, 0.89 against Claude Opus). Shares are lower bounds: referees reported the most consequential issues within 400 words.
| Major issue | Papers | Apr–Jun | Jul–Oct |
|---|---|---|---|
| Not making the materials needed for reproduction accessible | 51% | 49% | 52% |
| Claiming a cause without isolating it | 30% | 40% | 23% |
| Comparing against missing or unequal baselines | 17% | 17% | 17% |
| Not establishing repeatability or statistical uncertainty | 14% | 21% | 11% |
| Not checking that the scientific outputs are correct | 14% | 21% | 11% |
| Not specifying how the reported study was run | 13% | 18% | 9% |
| Measuring a proxy that does not establish the claimed success | 12% | 14% | 11% |
| Extending conclusions beyond the tested conditions | 10% | 9% | 10% |
| Not keeping evaluation independent of development | 9% | 10% | 8% |
| Reporting results or procedures that disagree | 8% | 14% | 4% |
| Using reference answers or judges without independent validation | 8% | 7% | 9% |
| Making a physics, mathematics, or statistical error | 8% | 13% | 4% |
Getting better, disclosing more
Does anyone read them?
Of the 84 papers submitted from 18 August, each preceded by about 185 papers in the set, 24 cite any of them, excluding probable self-citations. Of 9,992 references in HTML and source bibliographies, 240 (2.4%) point to an arXiv preprint from the window.
How reliable are the measurements?
| Check | Result |
|---|---|
| Same 20 papers, fresh referee | scores identical for 16–19 of 20, all within one point |
| Shuffled re-review of 20 early and late papers | the early–late gap persists (claims score 0.6 → 0.5 points); the gap in major-issue counts shrinks (−0.7 → −0.2) |
| Abstract labels, Haiku against a blind Sol recode | κ = 0.83 (use), 0.86 (scope), 0.63 (autonomy) |
| Issue taxonomy, Sol against Opus | κ = 0.89 |
All judgements come from language models with fixed prompts (in this repository). Per-paper scores and issues are not published; the review texts are committed to by SHA-256 (list, bundles).
Data
- aggregates.json: all aggregate results.
- papers.json: factual fields of the 228 reviewed papers (models, harness, code links, dates, AI-use statement, citation links, subject codes).
- Dataset: every screened record with its labels, searchable; records.json. Deposits on general-purpose repositories (Zenodo, figshare, OSF projects, no DOI) are not part of the dataset.
- Code, prompts and the query register: see the repository README.