MCiteBench:一个用于生成带引用文本的多模态基准
MCiteBench: A Multimodal Benchmark for Generating Text with Citations
AI总结:
针对多模态大语言模型的幻觉问题及现有引用生成研究多聚焦纯文本的不足,提出首个多模态引用生成基准MCiteBench,实验发现模型存在模态锚定困难与系统性模态偏差,为相关研究提供方向指引。
摘要:
多模态大语言模型(MLLMs)在整合多种模态方面取得了进展,但经常出现幻觉问题。缓解该问题的一个有前景的方案是生成带引用的文本,为验证提供透明的链条。然而,现有研究主要聚焦于纯文本内容的引用生成,多模态场景下的挑战在很大程度上仍未被探索。本文提出了MCiteBench,这是首个用于评估MLLMs在多模态语境下生成带引用文本能力的基准。该基准的数据来源于学术论文以及审稿-反驳互动内容,具备多样的信息来源和多模态内容。实验结果表明,MLLMs在处理多模态输入时难以可靠地锚定其输出。进一步分析发现了系统性的模态偏差,并揭示了模型在生成引用时内部如何依赖不同的信息来源,为理解模型行为提供了洞见,并为多模态引用任务的未来方向提供了指导。
英文摘要:
Multimodal Large Language Models (MLLMs) have advanced in integrating diverse modalities but frequently suffer from hallucination. A promising solution to mitigate this issue is to generate text with citations, providing a transparent chain for verification. However, existing work primarily focuses on generating citations for text-only content, leaving the challenges of multimodal scenarios largely unexplored. In this paper, we introduce MCiteBench, the first benchmark designed to assess the ability of MLLMs to generate text with citations in multimodal contexts. Our benchmark comprises data derived from academic papers and review-rebuttal interactions, featuring diverse information sources and multimodal content. Experimental results reveal that MLLMs struggle to ground their outputs reliably when handling multimodal input. Further analysis uncovers a systematic modality bias and reveals how models internally rely on different sources when generating citations, offering insights into model behavior and guiding future directions for multimodal citation tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。