Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Ant Group(蚂蚁集团);Alibaba Group(阿里巴巴集团)
Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding. However, existing studies have concentrated mainly on visual-textual misalignment, leaving largely unexplored the MLLMs' ability to preserve an originally correct answer when confronted with misleading information. We reveal a response uncertainty phenomenon: across nine standard datasets, twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue. To systematically quantify this vulnerability, we propose a two-stage evaluation pipeline: (1) elicit each model's original response on unperturbed inputs; (2) inject explicit (false-answer hints) and implicit (contextual contradictions) misleading instructions, and compute the misleading rate - the fraction of correct-to-incorrect flips. Leveraging the most susceptible examples, we curate the Multimodal Uncertainty Benchmark (MUB), a collection of image-question pairs stratified into low, medium, and high difficulty based on how many of twelve state-of-the-art MLLMs they mislead. Extensive evaluation on twelve open-source and five closed-source models reveals a high uncertainty: average misleading rates exceed 86%, with explicit cues over 67.19% and implicit cues over 80.67%. To reduce the misleading rate, we then fine-tune all open-source MLLMs on a compact 2000-sample mixed-instruction dataset, reducing misleading rates to 6.97% (explicit) and 32.77% (implicit), boosting consistency by nearly 29.37% on highly deceptive inputs, and slightly improving accuracy on standard benchmarks. Our code is available at https://github.com/Yunkaidang/uncertainty
Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion
Shenzhen International Graduate School Tsinghua University(清华大学深圳国际研究生院);Peng Cheng Laboratory(鹏城实验室);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Seoul National University(首尔大学)
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)
Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios
School of Computer Science and Engineering, Northwestern Polytechnical University(计算机科学与工程学院,西北工业大学);School of Computer Science and Technology, Xidian University(计算机科学与技术学院,西安电子科技大学);School of Artificial Intelligence and Big Data, Henan University of Technology(人工智能与大数据学院,河南工业大学)
From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
Kyoto University(京都大学);Georgia Institute of Technology(佐治亚理工学院);National Taiwan Normal University(台湾师范大学);Indiana University(印第安纳大学);Hong Kong University of Science(香港科学大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Cornell University(康奈尔大学);University of Liverpool(利物浦大学);University of Edinburgh(爱丁堡大学);Zhejiang University(浙江大学);Purdue University(Purdue 大学);Emory University, Atlanta, GA, USA(埃默里大学);West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China(西京生物大数据中心,四川大学西京医院,成都,中国)
Exploring the Design Space of Visual Context Representation in Video MLLMs
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Baichuan Inc.(百川公司);School of Information, Renmin University of China(中国人民大学信息学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Beijing University of Posts and Telecommunications(北京邮电大学);Zhongguancun Academy(中关村学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Chinese University of Hong Kong(香港中文大学);Xi’an Jiaotong University(西安交通大学)
Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
Nanyang Technological University(南洋理工大学);Northeastern University(东北大学);Chongqing University(重庆大学);University of Chinese Academy of Sciences(中国科学院大学);Sun Yat-sen University(中山大学)
School of Informatics, Xiamen University, China(厦门大学信息学院);WeChat AI, Tencent Inc, China(腾讯公司微信AI部门);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室)
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University(浙江大学 ZIP 实验室);South China University of Technology(华南理工大学);Pazhou Lab(琶洲实验室);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(大数据与智能机器人教育部重点实验室)
Revealing Multimodal Causality with Large Language Models
University of Technology Sydney(技术大学悉尼大学);Tongji University(同济大学);University of Melbourne(墨尔本大学);University of Sydney(悉尼大学);Macquarie University(麦考瑞大学)
ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
University of Science and Technology of China(中国科学技术大学);Huawei Technologies Ltd.(华为技术有限公司);State Key Laboratory of Communication Content Cognition, People’s Daily Online(人民网传播内容认知国家重点实验室)
Can Multimodal Large Language Models Truly Understand Small Objects?
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家级重点实验室);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);MiLM Plus;Xiaomi Inc(小米公司);Zhongguancun Academy(中关村学院);Beijing, China(北京市)
MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);MMLab, CUHK(CUHK多模态实验室)