arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00705cs.CV
收藏

无训练不确定性引导的复杂视觉任务多模态大语言模型

Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs

Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Helmholtz Munich(海德堡-慕尼黑亥姆霍兹研究中心);Google(谷歌);LTCI, Télécom Paris, Institut Polytechnique de Paris(巴黎理工大学 LTCI 实验室)

Sanghwan Kim, Rui Xiao, Stephan Alaniz, Yongqin Xian, Zeynep Akata

AI总结:

提出无需训练的框架,利用MLLM内在不确定性主动引导,通过响应不确定性评分候选视觉输入,使模型自主聚焦关键信息,在视觉搜索、长视频理解等任务中达到与微调系统相当的性能。

摘要:

多模态大语言模型(MLLMs)通常在细粒度感知方面存在困难,例如在高分辨率图像中识别小物体或在长视频中检测关键时刻。现有方法通常依赖于复杂的、任务特定的微调,这降低了泛化能力并增加了系统复杂性。在这项工作中,我们提出了一个有效的、无需训练的框架,利用MLLM的内在不确定性作为主动引导。我们的核心见解是,当提供相关视觉信息时,模型的不确定性会降低。我们引入了一个统一的机制,通过响应不确定性对候选视觉输入进行评分,使模型能够自主聚焦于信息量最大的数据。我们将这一简单原理应用于三个具有挑战性的视觉任务:视觉搜索、长视频理解和时间定位,使得现成的MLLMs能够达到与专门的、微调系统竞争的性能。我们的结果表明,利用内在不确定性是提高细粒度多模态性能的强大策略。

英文摘要:

Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or detecting key moments in long videos. Existing methods typically rely on complex, task-specific fine-tuning, which reduces generalizability and increases system complexity. In this work, we propose an effective, training-free framework that uses an MLLM's intrinsic uncertainty as proactive guidance. Our core insight is that a model's uncertainty decreases when provided with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most informative data. We apply this simple principle to three challenging visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned systems. Our results demonstrate that leveraging intrinsic uncertainty is a powerful strategy for improving fine-grained multimodal performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑