arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.11246cs.CL
收藏

像人类考官一样评判:一种用于生成任务的加权重要性多点评估框架

Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers

Pengcheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);Qwen team, Alibaba Group(通义实验室,阿里巴巴集团);Institute of Computing Technology, CAS(中国科学院计算技术研究所)

Guoxin Yu, Chulun Zhou, Lemao Liu, Qi Wang, Mo Yu, Jialong Tang, Baosong Yang, Xiang Ao, Wai Lam, Yue Yu

AI总结:

本文提出WIMPE框架,通过加权重要性多点评估方法,解决生成任务中长答案评估的挑战,提升与人类标注的关联性。

摘要:

在生成任务中评估模型响应的质量仍然具有挑战性,因为预期答案通常包含多个语义上不同但互补的因素,需要分解以实现细粒度评估。最近的评估方法依赖于任务级评分标准或问题感知的检查清单。然而,它们仍面临两个问题:1)难以评估响应是否真正基于提供的上下文;2)无法捕捉参考答案不同方面的异质重要性。受人类考官启发,我们提出了加权重要性多点评估(WIMPE)框架,将每个参考答案分解为加权的上下文绑定评分点。设计了两个互补的度量标准,即加权点对齐(WPA)和点对冲突惩罚(PCP),用于衡量模型响应与参考答案之间的对齐和矛盾。在10个生成任务上的大量实验表明,WIMPE在与人类标注的相关性上表现更高。

英文摘要:

Evaluating the quality of model responses remains challenging in generative tasks with long-form answers, as the expected answers usually contain multiple semantically distinct yet complementary factors that should be factorized for fine-grained assessment. Recent evaluation methods resort to relying on either task-level rubrics or question-aware checklists. However, they still 1) struggle to assess whether a response is genuinely grounded in provided contexts; 2) fail to capture the heterogeneous importance of different aspects of reference answers. Inspired by human examiners, we propose a Weighted Importance Multi-Point Evaluation (WIMPE) framework, which factorizes each reference answer into weighted context-bound scoring points. Two complementary metrics, namely Weighted Point-wise Alignment (WPA) and Point-wise Conflict Penalty (PCP), are designed to measure the alignment and contradiction between model responses and reference answers. Extensive experiments on 10 generative tasks demonstrate that WIMPE achieves higher correlations with human annotations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑