基于专家偏好的自动相关工作生成评估
Expert Preference-based Evaluation of Automated Related Work Generation
AI总结：
提出GREP多轮评估框架，结合经典标准与专家偏好，通过细粒度分解和对比示例提升LLM评估相关工作质量的能力，与人类专家评估高度相关。
摘要：
专家领域写作，如科学写作，通常需要广泛的领域知识。尽管大型语言模型（LLM）在此任务中显示出有前景的潜力，但评估自动生成的科学写作质量是一个关键开放问题，因为它需要领域特定标准的知识和辨别专家偏好的能力。传统的自动评估指标和LLM-as-a-judge系统主要为主流NLP任务设计，不足以把握专家偏好和领域特定的质量标准。为弥补这一差距并支持现实的人机协作写作，我们以相关工作生成为例，这是最具挑战性的科学任务之一。我们提出GREP，一个多轮评估框架，将经典相关工作评估标准与专家特定偏好相结合。GREP将评估分解为更细粒度的维度。这种局部评估进一步通过对比示例增强，为评估维度提供详细的上下文指导。实证研究表明，与标准LLM评判者相比，GREP能够以更稳健的方式评估相关工作章节的质量，反映科学写作的自然场景，并与人类专家的评估高度相关。我们还观察到，最先进的LLM生成的文本难以满足合适相关工作章节的验证约束。
英文摘要：
Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. GREP decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that GREP is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art LLMs struggle to satisfy validation constraints of a suitable related work section.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
