评估与增强大型语言模型在学术出版物中的新颖性评估能力
Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications
AI总结：
本文提出学术新颖性基准SchNovel和模拟人类审稿的RAG-Novelty方法，评估并增强了LLMs在学术论文新颖性评估中的能力。
摘要：
近期的研究主要从语义视角，使用认知科学的基准来评估大型语言模型（LLMs）的创造力/新颖性。然而，评估学术出版物中的新颖性在评估LLMs方面是一个很大程度上未被探索的领域。在本文中，我们引入了一个学术新颖性基准（SchNovel）来评估LLMs评估学术论文新颖性的能力。SchNovel包含从arXiv数据集中采样的六个领域的15000对论文，发表日期跨度为2到10年。在每一对中，假设最近发表的论文更具新颖性。此外，我们提出了RAG-Novelty，它通过利用相似论文的检索来评估新颖性，从而模拟人类审稿人采取的审稿过程。大量实验提供了关于不同LLMs评估新颖性能力的见解，并证明RAG-Novelty优于最近的基线模型。
英文摘要：
Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
