MDSEval：面向多模态对话摘要的元评估基准
MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization
AI总结：
针对多模态对话摘要领域缺乏可靠元评估基准的问题，该研究提出首个MDS元评估基准MDSEval，设计跨模态MEKI过滤框架保障数据质量，首次形式化MDS专属评估维度，基准测试揭示了现有评估方法的局限性。
摘要：
多模态对话摘要（Multimodal Dialogue Summarization，MDS）是一项应用广泛的关键任务。为助力开发高效的MDS模型，亟需可靠的自动评估方法以降低成本、减少人力投入，但这类方法需要依托人工标注构建的高质量元评估基准作为支撑。本文提出首个面向MDS的元评估基准MDSEval，包含含图片分享的对话、对应摘要，以及覆盖8个明确定义的质量维度的人工评判结果。为保障数据质量与丰富度，我们提出一种利用跨模态互斥关键信息（Mutually Exclusive Key Information，MEKI）的新型过滤框架。本研究首次识别并形式化了MDS特有的核心评估维度。我们对当前最优的模态评估方法进行了基准测试，发现这类方法难以区分先进多模态大语言模型（MLLMs）生成的摘要，且易受多种偏差影响。
英文摘要：
Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods are essential for reducing both cost and human effort. However, such methods require a strong meta-evaluation benchmark grounded in human annotations. In this work, we introduce MDSEval, the first meta-evaluation benchmark for MDS, consisting image-sharing dialogues, corresponding summaries, and human judgments across eight well-defined quality aspects. To ensure data quality and richfulness, we propose a novel filtering framework leveraging Mutually Exclusive Key Information (MEKI) across modalities. Our work is the first to identify and formalize key evaluation dimensions specific to MDS. We benchmark state-of-the-art modal evaluation methods, revealing their limitations in distinguishing summaries from advanced MLLMs and their susceptibility to various bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
