这个想法是否新颖？一种自动化的判断研究想法新颖性的基准
Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
AI总结：
本文提出RINoBench基准，用于评估研究想法新颖性判断，发现LLM生成的推理与人类相似但判断不准确。
摘要：
判断研究想法的新颖性对于推动科学发展至关重要，它能够识别未探索的方向，并确保贡献有意义地扩展现有知识而不是重复微小的变体。然而，鉴于科学文献的指数级增长，通过文献综述手动判断研究想法的新颖性是劳动密集型的、主观的，并且在大规模上不可行。因此，最近的研究提出了自动化的方法来判断研究想法的新颖性。然而，对这些方法的评估仍然很大程度上不一致，通常基于非标准化的人类评估，阻碍了大规模、可比的评估。为此，我们引入了RINoBench，这是首个全面的基准，用于大规模评估研究想法新颖性的判断。它包含1,381个研究想法，这些研究想法源自并由人类专家判断，以及九个自动化评估指标，用于评估基于标准的新颖性评分和新颖性判断的文本说明。使用这个基准，我们评估了几种最先进的大型语言模型（LLMs）在判断研究想法新颖性方面的能力。我们的发现表明，尽管LLM生成的推理与人类推理相似，但这种一致性并不能可靠地转化为准确的新颖性判断，这些判断与人类黄金标准判断显著不同，即使在领先的推理能力模型中也是如此。数据和代码可在：https://github.com/TimSchopf/RINoBench。
英文摘要：
Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
