MME-Unify：面向统一多模态理解与生成模型的综合基准
MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models
AI总结：
本文提出MME-Unify基准，通过标准化传统任务评估、五个统一多模态推理任务及12个U-MLLM与专用模型的系统对比，揭示了现有统一多模态模型在混合模态任务上的显著性能差距。
摘要：
现有的多模态大语言模型（MLLM）基准在评估统一多模态大语言模型（U-MLLM）时面临重大挑战，原因在于：1）缺乏针对传统任务的标准化基准，导致比较结果不一致；2）缺乏混合模态生成基准，无法评估多模态推理能力。我们提出了一个旨在系统评估U-MLLM的综合评估框架。我们的基准包括：标准化传统任务评估。我们从12个数据集中采样，覆盖10个任务、30个子任务，确保跨研究的一致和公平比较。统一任务评估。我们引入了五个测试多模态推理的新任务，包括图像编辑、带图像生成的常识问答和几何推理。综合模型基准测试。我们评估了12个领先的U-MLLM，如Janus-Pro、EMU3、VILA-U和Gemini2-flash，以及专门的模型，包括理解模型（如Claude-3.5-Sonnet）和生成模型（如DALL-E-3）。我们的发现揭示了现有U-MLLM存在显著的性能差距，突显了对能够有效处理混合模态任务的更稳健模型的需求。代码和评估数据可在https://mme-unify.github.io/找到。
英文摘要：
Existing MLLM benchmarks face significant challenges in evaluating Unified MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional tasks, leading to inconsistent comparisons; 2) absence of benchmarks for mixed-modality generation, which fails to assess multimodal reasoning capabilities. We present a comprehensive evaluation framework designed to systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30 subtasks, ensuring consistent and fair comparisons across studies." 2. Unified Task Assessment. We introduce five novel tasks testing multimodal reasoning, including image editing, commonsense QA with image generation, and geometric reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs, such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3). Our findings reveal substantial performance gaps in existing U-MLLMs, highlighting the need for more robust models capable of handling mixed-modality tasks effectively. The code and evaluation data can be found in https://mme-unify.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
