CoMet：多模态不确定性估计的上下文与多样性分解
CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation
AI总结：
提出CoMet方法，将多模态大模型的不确定性分解为上下文项和多样性项，通过轻量级后验模块高效估计，无需自回归生成或重复采样，在多个基准上优于现有方法。
摘要：
不确定性估计一直是AI模型中的长期挑战；它相当于“知道自己不知道”，而元认知即使对人类来说也是出了名的困难（参见邓宁-克鲁格效应）。尽管即使在更简单的分类系统中也远未解决，但在多模态大语言模型（MLLMs）中解决这一问题正变得越来越重要。在MLLMs中，不确定性可能源于多种来源及其关系，还可能源于开放设定中无界答案。为解决这些问题，我们提出CoMet，一种通过将不确定性分解为上下文特定项和多样性特定项来进行MLLM不确定性估计的方法。前者捕获给定上下文（例如任务或提示）引起的歧义，而后者捕获由上下文确定的合理答案中有多少与给定输入兼容。我们训练一个轻量级的后验不确定性模块来估计这些量，从而无需自回归答案生成或重复采样即可实现高效的不确定性估计。在各种开放多模态基准、幻觉检测和多项选择视觉问答基准上的实验表明，CoMet在保持实际效率的同时，持续改进了现有基线的不确定性估计。代码可在该https URL获取。
英文摘要：
Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it is still far from solved even in simpler classification systems, tackling it in multimodal large language models (MLLMs) is becoming increasingly important. Within MLLMs, uncertainty can stem from any of the diverse sources as well as from their relationships, and further can stem from the unbounded answers in the open-ended setting. To tackle the issues, we propose CoMet, an MLLM uncertainty estimation method by decomposing uncertainty into a context-specific term and a multiplicity-specific term. The former captures ambiguity induced by the given context (e.g., task or prompt), while the latter captures how many plausible answers determined by the context remain compatible with the given input. We train a lightweight post-hoc uncertainty module to estimate these quantities, which enables efficient uncertainty estimation without autoregressive answer generation or repeated sampling. Experiments on various open-ended multimodal benchmarks, hallucination detection, and multiple-choice visual question answering benchmarks show that CoMet consistently improves uncertainty estimation over existing baselines while remaining efficient in practice. Code is available at https://github.com/princetonvisualai/comet_uncertainty
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
