AboutAI
Hugging Face Blog
  • Score 94
  • Official

Can we trust LLMs to judge whether a research idea is novel?

Ai2 benchmark shows LLM novelty judges are brittle: small prompt changes swing accuracy by over 50 points, and simple prompted Claude Opus 4.6 beats purpose-bui

So what

What this event means by reading role—not a longer recap.

  • BuilderIf you use an LLM to score idea novelty, test prompt sensitivity and prefer a simple prompted baseline with majority voting over costly retrieval pipelines.
  • ResearcherNovelty-judge results are highly prompt-dependent, so report prompt variants and validate judges on AI-generated ideas, not just human papers.
  • InvestorAi2's benchmark and data suggest automated research-idea evaluation is still immature, tempering near-term claims around AI scientist products.
Read originalBack to Stories

Score dimensions

Higher total means read first. Each bar is one factor we use to rank the system pool. How we score

RelevanceHow tightly this is about AI.
88
ImpactHow much this could change the field or the market.
72
NoveltyHow new this is versus a recap.
80
CredibilityHow much we trust the source.
100
ActionabilityWhether a reader can do something with it.
65