VLM-UQBench：一种用于视觉语言模型中模态特定和跨模态不确定性的基准
VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
AI总结：
VLM-UQBench通过评估不同UQ方法在模态特定和跨模态不确定性上的表现，揭示了现有方法在细粒度不确定性检测上的不足。
摘要：
不确定性量化（UQ）对于确保视觉语言模型（VLMs）行为安全可靠至关重要。一个核心挑战是将不确定性定位到其源头，确定它是否源于图像、文本或两者之间的不一致。我们引入了VLM-UQBench，这是一个用于VLMs中模态特定和跨模态数据不确定性的基准，它由600个来自VizWiz数据集的真实世界样本组成，经过整理后分为清洁的图像、文本和跨模态不确定性子集，并配备一个可扩展的扰动管道，包含8种视觉、5种文本和3种跨模态扰动。我们进一步提出了两个简单的指标，用于量化UQ分数对这些扰动的敏感性以及与幻觉的相关性，并利用它们在四个VLM和三个数据集上评估了一系列UQ方法。实证研究表明：（i）现有UQ方法表现出强烈的模态特定专业化和对底层VLM的显著依赖；（ii）模态特定不确定性经常与幻觉同时出现，而当前UQ分数仅提供弱且不一致的风险信号；（iii）尽管UQ方法可以与基于推理的链式思考基线在明显、群体层面的不确定性上竞争，但它们在我们扰动管道引入的细微、实例层面的不确定性上大多失败。这些结果突显了当前UQ实践与可靠VLM部署所需的细粒度、模态感知不确定性之间的显著差距。
英文摘要：
Uncertainty quantification (UQ) is vital for ensuring that vision-language models (VLMs) behave safely and reliably. A central challenge is to localize uncertainty to its source, determining whether it arises from the image, the text, or misalignment between the two. We introduce VLM-UQBench, a benchmark for modality-specific and cross-modal data uncertainty in VLMs, It consists of 600 real-world samples drawn from the VizWiz dataset, curated into clean, image-, text-, and cross-modal uncertainty subsets, and a scalable perturbation pipeline with 8 visual, 5 textual, and 3 cross-modal perturbations. We further propose two simple metrics that quantify the sensitivity of UQ scores to these perturbations and their correlation with hallucinations, and use them to evaluate a range of UQ methods across four VLMs and three datasets. Empirically, we find that: (i) existing UQ methods exhibit strong modality-specific specialization and substantial dependence on the underlying VLM, (ii) modality-specific uncertainty frequently co-occurs with hallucinations while current UQ scores provide only weak and inconsistent risk signals, and (iii) although UQ methods can rival reasoning-based chain-of-thought baselines on overt, group-level ambiguity, they largely fail to detect the subtle, instance-level ambiguity introduced by our perturbation pipeline. These results highlight a significant gap between current UQ practices and the fine-grained, modality-aware uncertainty required for reliable VLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
