🔍 Read the full analysis: 5 Questions To Ask Before Trusting An LLM’s Novelty Assessment on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models produced unstable judgments of scientific idea novelty across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points; retrieval and extra reasoning did not consistently help. The team released a benchmark, data, code and an evaluation pipeline, while warning that some labels are imperfect.
Researchers at the Allen Institute for AI (Ai2) report, in the original analysis, that six large language models gave unstable assessments of scientific idea novelty when researchers changed evaluation details. In one pairwise experiment, a few words of prompt changes shifted accuracy by more than 50 percentage points. The team has released a benchmark, data, code and a pipeline to support further testing of systems intended to screen research ideas.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, as well as Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers used both absolute ratings, in which a model assesses an idea on its own, and pairwise comparisons, in which it judges one idea against another. The authors also report that adding literature retrieval or allowing more reasoning did not consistently improve results.
One benchmark set drew on ICLR 2026 peer reviews. It includes 154 ideas from accepted papers in the top-rated 10% of their research areas, where reviewers praised originality and none disputed it. Its lower-novelty group includes 145 ideas from rejected papers in the bottom 10%, selected where reviewers raised originality concerns and none praised it. An annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say the three disagreements did not change review classifications.
A second test setting paired the 154 human-authored ideas with 154 proposals from a basic language-model generator. The authors treated generated proposals as less novel, but call those labels weak because each idea was not individually verified. In an expert check of 30 pairs, selected heavily from cases where the strongest judge was wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Prompt Sensitivity Matters
Research groups may use LLMs to filter candidate ideas before spending time on literature searches, experiments or expert review. If a model’s judgment changes substantially after a small prompt edit, its score may reflect evaluation choices as well as the idea itself. That makes it harder to compare proposals across teams or to treat a single score as dependable evidence of originality.
The results also matter to efforts to automate parts of scientific discovery. Systems that generate hypotheses or summarize research need credible ways to judge whether a proposal adds something new. The study does not establish that every LLM novelty assessment fails; it shows that a score from one chosen setup is not, by itself, proof of reliable performance. Researchers and users need to know how a judge was tested and whether its findings hold under reasonable changes.
Popular right now
De’Longhi Classic Matte Black Espresso Machine with Milk Frother
Breville Barista Pro Espresso Machine with Grinder, Brushed Stainless Steel
Roborock Robot Vacuum and Mop with LiDAR Navigation and 10,000 Pa Suction
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Test Sets
In this study, a novelty judge is a model asked to assess whether an idea is new relative to existing work, either on its own or in comparison with another idea. The authors say there is no standard recipe: evaluations can differ in the model, prompt, comparison format, reasoning budget and access to retrieved literature. They also argue that judges are often tested on human-written papers even when a likely use is evaluating machine-generated proposals.
The benchmark uses reviewer comments as evidence about originality, rather than treating acceptance or rejection alone as a novelty label. The authors note that papers can be accepted or rejected for reasons beyond originality. They describe their work as part of a wider effort to develop systems that assist scientific discovery, including tools for research synthesis and hypothesis development. The released paper, data, code and automated pipeline are intended to let other researchers examine these choices and create additional evaluation sets.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels are signals of relative novelty, not a definitive ground truth. Reviewer comments and selection criteria support the human-idea categories, but they cannot establish originality in every case. The generated-idea comparison has a more direct limitation: the researchers assumed those proposals were less novel without verifying every one. The expert check covered only 30 pairs, not the complete set.
The supplied study description does not provide full results for every model, prompt, reasoning budget or evaluation format. It also does not show whether the reported instability generalizes to other academic fields, different idea generators or systems with broader access to research literature. The authors say some conclusions arise from disagreement between evaluation setups rather than benchmark labels, but the material provided does not detail how broadly that result held.
Further Tests Across Research Fields
Ai2 has made its benchmark, data, code and pipeline available for researchers to test evaluation choices and build further datasets. Other teams can examine whether prompt sensitivity appears in different disciplines, with different sources of ideas and under other judge designs, while testing approaches intended to make assessments more consistent.
The release description gives no specific follow-up study or timeline. For now, the results support caution about relying on one novelty score, while leaving open whether a particular combination of model, prompt, retrieval and evaluation method can produce dependable judgments.
Key Questions
What did the Ai2 researchers find?
They report that six tested LLMs’ scientific novelty judgments varied across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using absolute novelty ratings and pairwise comparisons.
Did literature retrieval or extra reasoning improve judgments?
The authors report that adding literature retrieval or more reasoning did not consistently improve the models’ judgments. The supplied material does not give complete results for every tested configuration.
Are the benchmark’s novelty labels definitive?
No. Human-idea labels rely on reviewer comments and selection criteria, while the generated-idea set assumes lower novelty without verifying each proposal individually. The authors describe some of these labels as weak rather than individually confirmed.
What did Ai2 release?
Ai2 released the study, benchmark data, code and an automated pipeline intended to help researchers build and test additional evaluation sets. No specific follow-up release date is given in the supplied material.
Primary source: Hugging Face · via ThorstenMeyerAI.com
