CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Hangzhou Dianzi University(杭州电子科技大学);School of Computer Science (National Pilot Software Engineering School), BUPT(计算机学院(国家级试点软件工程学院),北京邮电大学)
Yichen Yan, Ming Zhong, Qi Zhu, Xiaoling Gu, Jinpeng Chen, Huan Li
Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity.
We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling efficient and scalable data selection.
In our experiments, we trained the CoIDO scorer using only 20 percent of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20 percent subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2 percent of the performance of full-data fine-tuning, on average.
DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba Token Hub, Alibaba Group(阿里巴巴集团)
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机学院,北京大学);ByteDance(字节跳动)
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning
Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Peng Cheng Laboratory(鹏城实验室)
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);Tsinghua University(清华大学);Nanjing University(南京大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)
Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);Hunan Artificial Intelligence and Robotics Institute Company Ltd.(湖南人工智能与机器人研究院有限公司);Huawei Noah’s Ark Lab(华为诺亚实验室);Peng Cheng Laboratory(鹏城实验室)
Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regions
Department of Electronic Engineering and Computer Science University of California, Irvine(电子工程与计算机科学系 加州大学尔湾分校);Department of Computer Science University of Alabama at Birmingham(计算机科学系 阿拉巴马大学伯明翰分校)
RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation
MBZUAI(默罕默德·本·扎耶德人工智能大学);Shenzhen Campus of Sun Yat-Sen University(中山大学深圳校区);AAII, UTS(悉尼科技大学人工智能与创新研究所);University of Science and Technology of China(中国科学技术大学)
ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions
ByteDance Seed(字节跳动种子);University of Southern California(南加州大学);University of Tokyo(东京大学);University of California Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);University of California Los Angeles(加州大学洛杉矶分校)