arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.16813cs.AI
收藏

PeerArg:使用 LLM 进行论证式同行评审

PeerArg: Argumentative Peer Review with LLMs

Imperial College London(伦敦帝国学院)

Purin Sukpanichnant, Anna Rapberger, Francesca Toni

AI总结:

本文提出结合 LLM 与知识表示的 PeerArg 论证式同行评审系统,依据评审意见预测论文接收,并在三个数据集上验证其一种变体优于少样本端到端 LLM。

摘要:

同行评审是确定提交给科学会议或期刊论文质量的重要过程。然而,该过程具有主观性,并且容易受到偏见影响。已有若干研究应用 NLP 技术来支持同行评审,但这些研究基于黑盒技术,其输出难以解释和信任。本文提出一种用于支持并理解同行评审与决策过程的新型流水线:将 LLM 与知识表示方法相结合的 PeerArg 系统。PeerArg 接收一篇论文的一组评审意见作为输入,并输出论文的接收预测。我们在三个不同数据集上评估 PeerArg 流水线的性能,并将其与一种新型端到端 LLM 进行比较;该 LLM 使用少样本学习,根据评审意见预测论文是否被接收。结果表明,端到端 LLM 能够根据评审意见预测论文接收情况,但 PeerArg 流水线的一个变体优于该 LLM。

英文摘要:

Peer review is an essential process to determine the quality of papers submitted to scientific conferences or journals. However, it is subjective and prone to biases. Several studies have been conducted to apply techniques from NLP to support peer review, but they are based on black-box techniques and their outputs are difficult to interpret and trust. In this paper, we propose a novel pipeline to support and understand the reviewing and decision-making processes of peer review: the PeerArg system combining LLMs with methods from knowledge representation. PeerArg takes in input a set of reviews for a paper and outputs the paper acceptance prediction. We evaluate the performance of the PeerArg pipeline on three different datasets, in comparison with a novel end-2-end LLM that uses few-shot learning to predict paper acceptance given reviews. The results indicate that the end-2-end LLM is capable of predicting paper acceptance from reviews, but a variant of the PeerArg pipeline outperforms this LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑