GenCeption：使用无标签单模态数据评估视觉 LLMs
GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Data
AI总结：
本文提出无需标注的评估方法 GenCeption，通过迭代描述与生成量化跨模态语义漂移，建立 MMECeption 基准评估视觉 LLMs，验证了其有效性并揭示模型在文本密集任务上的不足。
摘要：
多模态大型语言模型（MLLMs）通常使用昂贵的标注多模态基准进行评估，而这些基准往往落后于快速发展的 MLLM 评估需求。本文概述并验证了 GenCeption，这是一种新颖的、无需标注的评估方法，仅需单模态数据即可测量跨模态语义连贯性，并反向评估 MLLMs 产生幻觉的倾向。该方法消除了昂贵数据标注的需求，将训练数据污染的风险降至最低，有望导致基准饱和更慢，并避免出现新兴能力的错觉。受 DrawCeption 游戏启发，GenCeption 从非文本样本开始，通过迭代描述和生成步骤进行。使用 GC@T 指标量化跨迭代的语义漂移。虽然 GenCeption 原则上适用于各种模态的 MLLMs，但本文重点关注其在视觉 LLMs（VLLMs）上的实现与验证。基于 GenCeption 方法，我们建立了用于评估 VLLMs 的 MMECeption 基准，并比较了几种流行 VLLMs 与人类标注者的表现。我们的实证结果验证了 GenCeption 的有效性，表明其与现有 VLLM 基准具有强相关性。VLLMs 仍显著落后于人类表现，尤其在文本密集型任务中表现不佳。
英文摘要：
Multimodal Large Language Models (MLLMs) are typically assessed using expensive annotated multimodal benchmarks, which often lag behind the rapidly evolving demands of MLLM evaluation. This paper outlines and validates GenCeption, a novel, annotation-free evaluation method that requires only unimodal data to measure inter-modality semantic coherence and inversely assesses MLLMs' tendency to hallucinate. This approach eliminates the need for costly data annotation, minimizes the risk of training data contamination, is expected to result in slower benchmark saturation, and avoids the illusion of emerging abilities. Inspired by the DrawCeption game, GenCeption begins with a non-textual sample and proceeds through iterative description and generation steps. The semantic drift across iterations is quantified using the GC@T metric. While GenCeption is principally applicable to MLLMs across various modalities, this paper focuses on its implementation and validation for Vision LLMs (VLLMs). Based on the GenCeption method, we establish the MMECeption benchmark for evaluating VLLMs, and compare the performance of several popular VLLMs and human annotators. Our empirical results validate GenCeption's effectiveness, demonstrating strong correlations with established VLLM benchmarks. VLLMs still significantly lag behind human performance and struggle especially with text-intensive tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
