多模态大海捞针：多模态大型语言模型的长上下文能力基准测试
Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
- Rutgers University（罗格斯大学）
- Microsoft Research（微软研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对多模态大型语言模型长上下文能力评估不足的问题，提出MMNeedle基准，通过多图像输入、图像拼接及自动标签生成协议评估模型，发现GPT-4o表现更优但负样本有幻觉，且API与开源模型存在性能差距。
AI中文摘要：
多模态大型语言模型（MLLMs）在各类应用中展现出显著潜力，引发了研究者和从业者的广泛关注。然而，对其长上下文能力的全面评估仍未得到充分探索。为填补这一空白，我们引入了多模态大海捞针（MMNeedle）基准，专门用于评估MLLMs的长上下文能力。除多图像输入外，我们采用图像拼接进一步增加输入上下文长度，并开发了自动生成子图像级检索标签的协议。本质上，MMNeedle通过压力测试MLLMs基于文本指令和图像内容描述定位目标子图像（针）的能力来评估它们，该设置需要对扩展的视觉上下文有深入理解，并能在长上下文图像输入中进行有效信息检索。利用该基准，我们评估了最先进的MLLMs，包括基于API和开源的模型。结果显示，GPT-4o在长上下文场景中持续优于其他模型，但在负样本（即针不在草堆中）中存在幻觉问题。我们对MLLMs的全面长上下文评估还揭示了基于API和开源模型之间存在显著的性能差距。所有重现主要结果所需的代码、数据和指令可在https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack获取。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown significant promise in various applications, leading to broad interest from researchers and practitioners alike. However, a comprehensive evaluation of their long-context capabilities remains underexplored. To address these gaps, we introduce the MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to assess the long-context capabilities of MLLMs. Besides multi-image input, we employ image stitching to further increase the input context length, and develop a protocol to automatically generate labels for sub-image level retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their capability to locate a target sub-image (needle) within a set of images (haystack) based on textual instructions and descriptions of image contents. This setup necessitates an advanced understanding of extensive visual contexts and effective information retrieval within long-context image inputs. With this benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and open-source models. The findings reveal that GPT-4o consistently surpasses other models in long-context scenarios, but suffers from hallucination problems in negative samples, i.e., when needles are not in the haystacks. Our comprehensive long-context evaluation of MLLMs also sheds lights on the considerable performance gap between API-based and open-source models. All the code, data, and instructions required to reproduce the main results are available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.
