超越文本的衡量：通过质量、对齐和多样性评估多模态摘要
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity
AI总结：
本文提出MM-Eval框架，整合文本质量、跨模态对齐和视觉多样性评估，通过学习聚合模型优化多模态摘要的综合评价，揭示事实一致性对整体质量的关键作用。
摘要：
多模态大语言模型（MLLMs）促进了多模态摘要（MSMO），其中系统生成简洁文本摘要并配以显著的多模态视觉内容。然而，当前MSMO评估仍显碎片化：文本质量、图像-文本对齐和视觉多样性通常孤立评估，难以捕捉模态间是否共同支持忠实有用的摘要。为解决这一差距，我们引入MM-Eval，统一评估框架整合文本质量、跨模态对齐和视觉多样性评估。MM-Eval包含三个组件：（1）文本质量，通过OpenFActScore评估事实一致性，通过G-Eval评估连贯性、流畅性和相关性；（2）图像-文本相关性，通过MLLM-as-a-judge方法评估；（3）图像集多样性，通过截断CLIP熵量化。我们通过在mLLM-EVAL新闻基准上训练的 learned aggregation 模型校准MM-Eval，对齐组件贡献与人类偏好。分析揭示了文本主导的层级结构，其中事实一致性是感知整体质量的关键决定因素，而视觉相关性和多样性提供互补信号。MM-Eval优于启发式聚合基线，提供了一个可解释、参考弱的多模态摘要比较评估框架。
英文摘要：
Multimodal Large Language Models (MLLMs) have facilitated Multimodal Summarization with Multimodal Output (MSMO), wherein systems generate concise textual summaries accompanied by salient visuals from multimodal sources. However, current MSMO evaluation remains fragmented: text quality, image-text alignment, and visual diversity are typically assessed in isolation using unimodal metrics, making it difficult to capture whether the modalities jointly support a faithful and useful summary. To address this gap, we introduce MM-Eval, a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. MM-Eval comprises three components: (1) text quality, measured using OpenFActScore for factual consistency and G-Eval for coherence, fluency, and relevance; (2) image-text relevance, evaluated via an MLLM-as-a-judge approach; and (3) image-set diversity, quantified using Truncated CLIP Entropy. We calibrate MM-Eval through a learned aggregation model trained on the mLLM-EVAL news benchmark, aligning component contributions with human preferences. Our analysis reveals a text-dominant hierarchy in this setting, where factual consistency acts as a critical determinant of perceived overall quality, while visual relevance and diversity provide complementary signals. MM-Eval improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
