作者参与的回信生成与评估:将作者专业知识和意图整合到对同行评审的回复中
Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review
AI总结:
提出作者参与的回信生成与评估框架,通过引入对齐的评审-回复-修订三元组数据集、支持灵活作者输入和可控生成的REspGen系统以及包含20+指标的综合评估套件REspEval,填补了作者信号利用和评估的空白。
摘要:
作者回复(反驳)写作是科学同行评审的关键阶段,需要作者付出大量努力。在实践中,作者拥有领域专业知识、仅作者可用的信息和回复策略——作者专业知识和意图的具体形式——并寻求NLP辅助,将这些信号整合到作者回复生成(ARG)中。然而,这种作者参与范式缺乏正式的NLP表述和系统研究:没有数据集提供细粒度的作者信号,现有的ARG工作缺乏作者输入和控制,也没有评估指标衡量回复对作者信号的反映以及解决评审者关注点的有效性。为填补这些空白,我们引入了(i)Re3Align,第一个大规模的对齐评审-回复-修订三元组数据集,其中修订代理作者信号;(ii)REspGen,一个作者参与的ARG框架,支持灵活的作者输入、多属性控制和评估引导的细化;以及(iii)REspEval,一个包含20多个指标的全面评估套件,涵盖输入利用、可控性、回复质量和话语。使用SOTA LLMs的实验证明了作者输入和评估引导细化的好处、输入特异性对回复质量的影响以及可控性与质量之间的权衡。我们发布了我们的数据集、生成和评估工具。
英文摘要:
Author response (rebuttal) writing is a critical stage of scientific peer review that demands substantial author effort. In practice, authors possess domain expertise, author-only information, and response strategies - concrete forms of author expertise and intent - and seek NLP assistance that integrates these signals into author response generation (ARG). Yet this author-in-the-loop paradigm lacks formal NLP formulation and systematic study: no dataset provides fine-grained author signals, existing ARG work lacks author inputs and controls, and no evaluation measures response reflection of author signals and effectiveness in addressing reviewer concerns. To fill these gaps, we introduce (i) Re3Align, the first large-scale dataset of aligned review-response-revision triplets, where revisions proxy author signals; (ii) REspGen, an author-in-the-loop ARG framework supporting flexible author input, multi-attribute control, and evaluation-guided refinement; and (iii) REspEval, a comprehensive evaluation suite with 20+ metrics spanning input utilization, controllability, response quality, and discourse. Experiments with SOTA LLMs demonstrate the benefits of author input and evaluation-guided refinement, the impact of input specificity on response quality, and controllability-quality trade-offs. We release our dataset, generation and evaluation tools.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。