Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Helmholtz Munich(海德堡-慕尼黑亥姆霍兹研究中心);Google(谷歌);LTCI, Télécom Paris, Institut Polytechnique de Paris(巴黎理工大学 LTCI 实验室)
Sanghwan Kim, Rui Xiao, Stephan Alaniz, Yongqin Xian, Zeynep Akata
Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or detecting key moments in long videos. Existing methods typically rely on complex, task-specific fine-tuning, which reduces generalizability and increases system complexity. In this work, we propose an effective, training-free framework that uses an MLLM's intrinsic uncertainty as proactive guidance. Our core insight is that a model's uncertainty decreases when provided with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most informative data. We apply this simple principle to three challenging visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned systems. Our results demonstrate that leveraging intrinsic uncertainty is a powerful strategy for improving fine-grained multimodal performance.
GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
City University of Hong Kong(香港城市大学);Huawei Research(华为研究院);University of Science and Technology of China(中国科学技术大学);Chinese University of Hong Kong(香港中文大学);City University of Hong Kong (Dongguan)(香港城市大学(东莞))
UR$^{2}$-MLLM: Uncertainty-aware Revisit Reasoning in Multimodal Large Language Models for Radiology Report Generation
MedVisAI Lab(MedVisAI实验室);Lee Kong Chian School of Medicine, Nanyang Technological University(南洋理工大学李光前医学院);Centre of AI in Medicine, Singapore(新加坡医学人工智能中心);AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)AI方向);Ruijin Hospital, Shanghai Jiao Tong University School of Medicine(上海交通大学医学院附属瑞金医院)
From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognition
College of Computer Science, Sichuan University(四川大学计算机学院);School of Artificial Intelligence, Sichuan University(四川大学人工智能学院);Tianfu Jincheng Laboratory(天府锦城实验室)
Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Zhongguancun Academy(中关村学院);Southeast Academy of Information Technology, Beijing Institute of Technology(北京理工大学信息科技东南学院);Zhongguancun Laboratory(中关村实验室)
HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts
Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学)
Backdoor Cleaning without External Guidance in MLLM Fine-tuning
School of Computer Science, Wuhan University(武汉大学计算机学院);Munich Research Center(慕尼黑研究中心);Huawei Technologies(华为技术有限公司);Nanyang Technological University(南洋理工大学)
LLM-Bootstrapped Targeted Finding Guidance for Factual MLLM-based Medical Report Generation
Zhejiang University(浙江大学);The Second Affiliated Hospital Zhejiang University School of Medicine(浙江大学医学院附属第二医院);Hangzhou Pu Jian Medical Technology Co., Ltd.(杭州普健医疗科技有限公司)
EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models
Shanghai AI Laboratory(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes
Integrated Vision and Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(整合视觉与语言实验室,电气工程学院,韩国科学技术院(KAIST))
Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detection
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute of AI for Industries(工业人工智能研究所);University of Chinese Academy of Sciences(中国科学院大学)
Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs
School of Information Science and Engineering, Lanzhou University(兰州大学信息科学与工程学院);Beijing University of Posts and Telecommunications(北京邮电大学);Cloud and AI BU, Huawei(华为云与AI业务部);School of Computing, National University of Singapore(新加坡国立大学计算机学院)
The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
Beijing Jiaotong University(北京交通大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国)
VALLR-Pin: Uncertainty-Factorized Visual Speech Recognition for Mandarin with Pinyin Guidance
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(网络与交换技术国家重点实验室,北京邮电大学);First Research Institute of the Ministry of Public Security of the People’s Republic of China(中华人民共和国公安部第一研究所)