多模态检索增强的多模态生成：数据集、评估指标与强基线
Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
AI总结：
本文系统研究多模态检索增强多模态生成（M$^2$RAG）任务，构建基准、提出评估指标与处理策略，微调7B-8B模型超越GPT-4o并接近o3-mini。
摘要：
我们对多模态检索增强的多模态生成（M$^2$RAG）这一新任务进行了系统性研究，该任务使基础模型能够处理多模态网页内容并生成多模态响应，展现出更好的信息密度和可读性。尽管其潜在影响巨大，M$^2$RAG 仍研究不足，缺乏全面分析和高质量数据资源。为弥补这一空白，我们通过严格的数据筛选流程建立了一个综合基准，并采用基于基础模型的文本模态指标和多模态指标进行评估。我们进一步提出了几种策略，使基础模型能有效处理 M$^2$RAG 任务，并通过使用我们设计的指标筛选高质量样本来构建训练集。我们的广泛实验证明了所提指标的可靠性，展示了在我们设计的策略下模型性能的全景，并表明我们微调的 7B-8B 模型优于 GPT-4o 模型，接近最先进的 OpenAI o3-mini。此外，我们跨多个领域进行了细粒度分析，并验证了数据筛选流程设计的有效性。所有资源，包括代码、数据集和模型权重，将公开发布。
英文摘要：
We present a systematic investigation of Multi-modal Retrieval Augmented Multi-modal Generation (M$^2$RAG), a novel task that enables foundation models to process multi-modal web content and generate multi-modal responses, which exhibits better information density and readability. Despite its potential impact, M$^2$RAG remains understudied, lacking comprehensive analysis and high-quality data resources. To address this gap, we establish a comprehensive benchmark through a rigorous data curation pipeline, and employ text-modal metrics and multi-modal metrics based on foundation models for evaluation. We further propose several strategies for foundation models to process M$^2$RAG task effectively and construct a training set by filtering high-quality samples using our designed metrics. Our extensive experiments demonstrate the reliability of our proposed metrics, a landscape of model performance within our designed strategies, and show that our fine-tuned 7B-8B models outperform the GPT-4o model and approach the state-of-the-art OpenAI o3-mini. Additionally, we perform fine-grained analyses across diverse domains and validate the effectiveness of our designs in data curation pipeline. All resources, including codes, datasets, and model weights, will be publicly released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
