Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Baichuan Inc.(百川智能公司);New Laboratory of Pattern Recognition (NLPR)(新模式识别实验室);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Han Huang, Yuqi Huo, Zijia Zhao, Haoyu Lu, Shu Wu, Bingning Wang, Qiang Liu, Weipeng Chen, Liang Wang
Multimodal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets. However, $\textit {de facto}$ filter-based data quality enhancement paradigms often discard a substantial portion of high-quality image data due to inadequate semantic alignment between images and texts, leading to inefficiencies in data utilization and scalability. In this paper, we propose the Adaptive Image-Text Quality Enhancer (AITQE), a model that dynamically assesses and enhances the quality of image-text pairs. AITQE employs a text rewriting mechanism for low-quality pairs and incorporates a negative sample learning strategy to improve evaluative capabilities by integrating deliberately selected low-quality samples during training. Unlike prior approaches that significantly alter text distributions, our method minimally adjusts text to preserve data volume while enhancing quality. Experimental results demonstrate that AITQE surpasses existing methods on various benchmark, effectively leveraging raw data and scaling efficiently with increasing data volumes. We hope our work will inspire future works. The code and model are available at: https://github.com/hanhuang22/AITQE.
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University(哈杰泰佩大学);Aselsan Research(阿塞尔桑研究院);Koç University(科克大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心,阿姆斯特丹大学);Amsterdam Public Health(阿姆斯特丹公共卫生研究院)
GeoAlign: Geometric Feature Realignment for MLLM Spatial Reasoning
National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(视觉技术国家工程研究中心、多媒体信息处理国家重点实验室、计算机学院、北京大学);Meituan Inc.(美团公司);National Biomedical Imaging Center, Peking University(生物医学成像国家中心、北京大学)
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong Polytechnic University(香港理工大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Hong Kong University of Science and Technology(香港科技大学);Apple(苹果公司);Peking University(北京大学);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学)
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Peng Cheng Laboratory(鹏城实验室);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室);Tsinghua Shenzhen International Graduate School(清华大学深圳国际 Graduate School);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Yinwang Intelligent Technology Co., Ltd.(亿纬智能科技有限公司);Huawei’s 2012 Lab(华为2012实验室)
SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding
OpenGVLab(开放视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);MMLab(多媒体实验室);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院);Shanghai Jiao Tong University(上海交通大学)
Backdoor Cleaning without External Guidance in MLLM Fine-tuning
School of Computer Science, Wuhan University(武汉大学计算机学院);Munich Research Center(慕尼黑研究中心);Huawei Technologies(华为技术有限公司);Nanyang Technological University(南洋理工大学)
ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Southeast University(东南大学);Beijing University of Posts and Telecommunications(北京邮电大学);National University of Singapore(新加坡国立大学);Wuhan AI Research(武汉人工智能研究院);Guangdong Provincial Key Laboratory of Intellectual Property and Big Data, Guangdong Polytechnic Normal University(广东技术师范大学广东省知识产权大数据重点实验室)
Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学,中国);ByteDance(字节跳动);Northeastern University(东北大学)
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University(京畿大学计算机学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);College of Computer Science and Information Engineering, Harbin Normal University(哈尔滨师范大学计算机科学与信息工程学院);College of Professional and Continuing Education, The Hong Kong Polytechnic University(香港理工大学专业及继续教育学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Corrupted but Not Broken: Understanding and Mitigating the Negative Impacts of Corrupted Data in Visual Instruction Tuning
Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Hong Kong Baptist University(香港 Baptist 大学)
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
The Hong Kong Polytechnic University(香港理工大学);Sun Yat-sen University(中山大学);InfiX.ai;PolyU-Daya Bay Technology and Innovation Research Institute(PolyU-大亚湾技术与创新研究院)
Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detection
Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen Key Laboratory of Media Security, and SZU-AFS Joint Innovation Center for AI Technology(广东省智能信息处理重点实验室、深圳媒体安全重点实验室及深圳大学-AFS联合人工智能技术创新中心);Tencent Youtu Lab(腾讯优图实验室);Peking University(北京大学);Sun Yat-Sen University(中山大学)
mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information(多模态信息超级智能安全北京市重点实验室);Tencent Inc.(腾讯公司);Huawei Noah’s Ark Laboratory(华为诺亚方舟实验室)