Multimodal Large Language Models (MLLMs) can enhance trustworthiness by aligning with human preferences. As human preference labeling is laborious, recent works employ evaluation models for assessing MLLMs' responses, using the model-based assessments to automate preference dataset construction. This approach, however, faces challenges with MLLMs' lengthy and compositional responses, which often require diverse reasoning skills that a single evaluation model may not fully possess. Additionally, most existing methods rely on closed-source models as evaluators. To address limitations, we propose DecompGen, a decomposable framework that uses an ensemble of open-sourced expert models. DecompGen breaks down each response into atomic verification tasks, assigning each task to an appropriate expert model to generate fine-grained assessments. The DecompGen feedback is used to automatically construct our preference dataset, DGPref. MLLMs aligned with DGPref via preference learning show improvements in trustworthiness, demonstrating the effectiveness of DecompGen.
Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives
School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳));Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Dermatology, The First Affiliated Hospital, Shantou University Medical College(皮肤科,汕头大学医学院第一附属医院)
DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness
Department of Computer Science(计算机科学系);Engineering, Ohio State University, Ohio, US(工程系,俄亥俄州立大学,俄亥俄,美国);Department of Computer Science, University of Central Florida, Florida, US(计算机科学系,中央佛罗里达大学,佛罗里达,美国)
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Department of Computer Science, Khalifa University of Science and Technology(卡利法科技大学计算机科学系);Information Technology University(信息技术大学);KAU(卡乌大学);University of the Western Australia(西澳大学)
CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
State Key Laboratory of Complex and Critical Software Environment, Beihang University(北京航空航天大学复杂关键软件环境国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DAMO Academy, Alibaba Group(阿里巴巴达摩院);Hupan Lab(湖畔实验室);Zhejiang University(浙江大学)
MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs
Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学);Guilin University of Electronic Technology(桂林电子科技大学)
DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM
Zhejiang University(浙江大学);Shanghai AI lab(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);The University of Sydney(悉尼大学);University of Oxford(牛津大学)
Tsinghua University(清华大学);BNRist;OPPO AI Center(OPPO人工智能中心);Peking University(北京大学);Hangzhou Zhuoxi Institute of Brain and Intelligence(杭州智行脑科学与人工智能研究所)
Cantor: Inspiring Multimodal Chain-of-Thought of MLLM
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(厦门大学 多媒体可信感知与高效计算重点实验室(教育部));Tencent Youtu Lab(腾讯优图实验室)
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Salesforce AI Research(Salesforce AI 研究院);University of Cambridge(剑桥大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Carnegie Mellon University(卡内基梅隆大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Peng Cheng Laboratory(鹏城实验室);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室);Tsinghua Shenzhen International Graduate School(清华大学深圳国际 Graduate School);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Yinwang Intelligent Technology Co., Ltd.(亿纬智能科技有限公司);Huawei’s 2012 Lab(华为2012实验室)
Visual Question Decomposition on Multimodal Large Language Models
Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Amazon Web Services(亚马逊云计算服务);LMU Munich(慕尼黑大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Oxford(牛津大学)
DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Prompt
SKLCCSE, Institute of Artificial Intelligence, Beihang University, Beijing, China(软件学院、人工智能研究院、北航、北京、中国);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing, China(未来区块链与隐私计算先进创新中心、北航、北京、中国)
PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);Beijing Jiaotong University(北京交通大学);ShanghaiTech University(上海科技大学)