RelevAI-Reviewer:一个面向综述论文相关性的 AI 审稿人基准
RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance
AI总结:
本文提出 RelevAI-Reviewer 基准,将综述论文与征稿提示的相关性评审建模为分类任务,并构建含 25,164 个实例的数据集;实验显示基于 BERT 的端到端分类器优于传统 ML 基线。
摘要:
人工智能(AI)的最新进展,尤其是大语言模型(LLM)的广泛采用,显著提升了文本分析能力。这一技术演进为自动化科学论文评审带来了可观前景;传统上,该任务由同行研究者通过同行评审完成。尽管传统同行评审在维持研究质量方面发挥着关键作用,但其过程往往缓慢且容易受到偏见影响,可能阻碍科学知识的快速传播。本文提出 RelevAI-Reviewer,这是一个自动化系统,将综述论文评审任务概念化为一个分类问题,旨在评估论文与指定提示(类似于“征稿启事”)之间的相关性。为解决该问题,我们引入了一个包含 25,164 个实例的新颖数据集。每个实例包含一个提示和四篇候选论文,每篇论文与该提示的相关性各不相同。目标是开发一个机器学习(ML)模型,使其能够判断每篇论文的相关性并识别最相关的一篇。我们探索了多种基线方法,包括支持向量机(SVM)等传统 ML 分类器,以及 BERT 等先进语言模型。初步结果表明,基于 BERT 的端到端分类器在性能上优于其他传统 ML 方法。我们将此问题作为一项公开挑战提出,以促进该研究领域的参与和兴趣。
英文摘要:
Recent advancements in Artificial Intelligence (AI), particularly the widespread adoption of Large Language Models (LLMs), have significantly enhanced text analysis capabilities. This technological evolution offers considerable promise for automating the review of scientific papers, a task traditionally managed through peer review by fellow researchers. Despite its critical role in maintaining research quality, the conventional peer-review process is often slow and subject to biases, potentially impeding the swift propagation of scientific knowledge. In this paper, we propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem, aimed at assessing the relevance of a paper in relation to a specified prompt, analogous to a "call for papers". To address this, we introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt. The objective is to develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one. We explore various baseline approaches, including traditional ML classifiers like Support Vector Machine (SVM) and advanced language models such as BERT. Preliminary findings indicate that the BERT-based end-to-end classifier surpasses other conventional ML methods in performance. We present this problem as a public challenge to foster engagement and interest in this area of research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。