arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08149cs.CLcs.AI
收藏

DIAL-SUMMER:一种评估对话摘要中分层错误的结构化框架

DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries

University of Southern California(美国南加州大学);Capital One

Sahana Ramnath, Nima Chitsazan, Mingyang Zhou, Chia-Hsuan Lee, Shi-Xiong Zhang, Stephen Rawls, Sambit Sahu, Sangwoo Cho, Xiang Ren, Genta Indra Winata, Akshaj Kumar Veldanda

AI总结:

DIAL-SUMMER提出一种结构化框架,用于评估对话摘要中分层错误,通过细粒度错误分类和实证分析揭示摘要生成的挑战。

摘要:

对话是人类的主要交流方式,自动生成对话摘要(例如,用于审查会议中讨论的关键点,或回顾客户代理与产品用户之间的对话)具有极大的帮助。先前的对话摘要评估工作大多忽略了这一任务的复杂性:(i)结构的转变,从多个说话者在多个回合中分散讨论信息,到摘要句子的结构;(ii)叙述视角的转变,从说话者的第一/第二人称叙述,到摘要中标准化的第三人称叙述。在本工作中,我们介绍了我们的框架DIALSUMMER来解决上述问题。我们提出了DIAL-SUMMER的错误分类法,以全面评估对话摘要的两个层次水平:对话层次,关注更广泛的说话者和回合,以及回合内层次,关注回合内讨论的信息。我们随后展示了由我们的分类法细粒度错误手动注释的DIAL-SUMMER数据集。我们对这些注释的错误进行了实证分析,并观察到有趣的趋势(例如,对话中间的回合最常被遗漏在摘要中,外部幻觉主要发生在摘要的末尾)。我们还通过实验测试了LLM-Judges在检测这些错误方面的能力,通过这我们展示了我们数据集的挑战性,我们分类法的鲁棒性,以及未来工作在该领域增强LLM性能的必要性。代码和推理数据集即将发布。

英文摘要:

Dialogues are a predominant mode of communication for humans, and it is immensely helpful to have automatically generated summaries of them (e.g., to revise key points discussed in a meeting, to review conversations between customer agents and product users). Prior works on dialogue summary evaluation largely ignore the complexities specific to this task: (i) shift in structure, from multiple speakers discussing information in a scattered fashion across several turns, to a summary's sentences, and (ii) shift in narration viewpoint, from speakers' first/second-person narration, standardized third-person narration in the summary. In this work, we introduce our framework DIALSUMMER to address the above. We propose DIAL-SUMMER's taxonomy of errors to comprehensively evaluate dialogue summaries at two hierarchical levels: DIALOGUE-LEVEL that focuses on the broader speakers/turns, and WITHIN-TURN-LEVEL that focuses on the information talked about inside a turn. We then present DIAL-SUMMER's dataset composed of dialogue summaries manually annotated with our taxonomy's fine-grained errors. We conduct empirical analyses of these annotated errors, and observe interesting trends (e.g., turns occurring in middle of the dialogue are the most frequently missed in the summary, extrinsic hallucinations largely occur at the end of the summary). We also conduct experiments on LLM-Judges' capability at detecting these errors, through which we demonstrate the challenging nature of our dataset, the robustness of our taxonomy, and the need for future work in this field to enhance LLMs' performance in the same. Code and inference dataset coming soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑