Skip to content
Hot eventDeveloping

Study finds LLM peer reviewers over-focus on paper appendices

1 reports1 reporting sourcesUpdated 49 min ago

Start here

AI digest

A new study, "AI Reviews Are Biased Towards the Appendix," reports that newer AI models draw most of their review criticisms from paper appendices rather than main findings. Testing 2,000 papers from ACL 2026, EMNLP 2026, COLM 2026 and ICLR 2027 with GPT-6-Astra, GPT-6.1-Sol, Claude-Opus-5.5 and Claude-Sonnet-5.5, the authors found AI reviewers attributed 62% of weaknesses to appendix material versus 8.4% for humans; newer models drew roughly 50–70% of criticisms from appendices, compared with under 30% for GPT-5. The bias persisted across Codex, API and web access. The authors also report that appendix-related criticism in human reviews rose to 27.6% in the latest reviews after years below 11%. For the tests, internet access was disabled so reviews relied only on the provided PDFs, and author identities were removed from the ACL, EMNLP and COLM papers. The study's code is available on GitHub.

Generated from reports · updated 44 min ago

Report timeline

Follow the reports to see the event from different sides.

November 16
  1. Hacker News · AI
    Study finds AI peer reviewers fixate on paper appendices over main findings

    A new study titled AI Reviews Are Biased Towards the Appendix found that newer AI models draw roughly 50-70% of their review criticisms from paper appendices, compared with under 30% for GPT-5. Testing 2,000 papers from ACL 2026, EMNLP 2026, COLM 2026 and ICLR 2027 with GPT-6-Astra, GPT-6.1-Sol, Claude-Opus-5.5 and Claude-Sonnet-5.5, the authors report AI reviewers attributed 62% of weaknesses to appendix material versus 8.4% for humans, and the bias persisted across Codex, API and web access. The paper also reports that appendix-related criticism in human peer reviews rose to 27.6% in the latest reviews after years below 11%, and its code is available on GitHub.

Heat for this event

Not enough continuous observations yet to draw a trend.