Defend:用于同行评审的自动化反驳与最小作者指导
Defend: Automated Rebuttals for Peer Review with Minimal Author Guidance
AI总结:
本文提出DEFEND工具,通过作者引导的结构化推理生成反驳,提升事实准确性与反驳力度,对比三种方法显示其有效性。
摘要:
反驳生成是科学论文同行评审的关键环节,使作者能澄清误解、纠正事实错误并引导评审更准确评估。我们发现大型语言模型(LLMs)在直接用于反驳生成时难以进行针对性反驳并保持准确的事实基础,凸显了结构化推理和作者干预的必要性。为此,本文介绍DEFEND工具,该工具基于LLM设计,明确执行自动化反驳生成的推理过程,同时保持作者参与。与直接编写反驳不同,作者只需驱动推理过程,实现高效且低认知负担的方法。我们比较DEFEND与三种方法:(i)直接使用LLM进行反驳(DRG),(ii)分段使用LLM进行反驳(SWRG),(iii)无作者干预的分段顺序方法(SA)。为实现细粒度评估,我们扩展ReviewCritique数据集,创建评审分段、缺陷、错误类型注释、反驳行动标签以及映射到黄金反驳段落。实验结果和用户研究显示,直接使用LLM在事实正确性和针对性反驳上表现不佳。分段生成和带有作者参与的自动化顺序方法显著提升了事实正确性和反驳力度。
英文摘要:
Rebuttal generation is a critical component of the peer review process for scientific papers, enabling authors to clarify misunderstandings, correct factual inaccuracies, and guide reviewers toward a more accurate evaluation. We observe that Large Language Models (LLMs) often struggle to perform targeted refutation and maintain accurate factual grounding when used directly for rebuttal generation, highlighting the need for structured reasoning and author intervention. To address this, in the paper, we introduce DEFEND an LLM based tool designed to explicitly execute the underlying reasoning process of automated rebuttal generation, while keeping the author-in-the-loop. As opposed to writing the rebuttals from scratch, the author needs to only drive the reasoning process with minimal intervention, leading an efficient approach with minimal effort and less cognitive load. We compare DEFEND against three other paradigms: (i) Direct rebuttal generation using LLM (DRG), (ii) Segment-wise rebuttal generation using LLM (SWRG), and (iii) Sequential approach (SA) of segment-wise rebuttal generation without author intervention. To enable finegrained evaluation, we extend the ReviewCritique dataset, creating review segmentation, deficiency, error type annotations, rebuttal-action labels, and mapping to gold rebuttal segments. Experimental results and a user study demonstrate that directly using LLMs perform poorly in factual correctness and targeted refutation. Segment-wise generation and the automated sequential approach with author-in-the-loop, substantially improve factual correctness and strength of refutation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。