Instruction tuning, or supervised finetuning on extensive task-specific data, is necessary for Large Vision-Language Models (LVLMs) to generalize well across a broad range of vision-language (VL) tasks. However, training on large VL datasets can become prohibitively expensive. In this work, we introduce COINCIDE, an effective and scalable data selection technique that uses a small model as a reference model to select visual instruction tuning data for efficient finetuning of a target LVLM, focusing on diversity and transferability. Specifically, we cluster the training data using internal activations from a small model, which identifies VL concept-skill compositions needed by a target LVLM. We then sample data from these diverse clusters by considering their density and transferability, or the ability to transfer well to other concept-skill compositions. This approach ensures the diversity of these compositions, which is vital for LVLM generalization. Extensive experiments demonstrate that COINCIDE achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan. Using only 20% of the LLaVA-1.5 dataset, COINCIDE achieves performance comparable to the LVLM finetuned on the whole dataset, with 70% reduction of the wall-clock running time. On the Vision-Flan dataset, our method achieves superior results with only 16.7% of the training data.
Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
Mehta Family School of Data Science and Artificial Intelligence(梅hta家族数据科学与人工智能学院);Indian Institute of Technology, Roorkee(印度理工学院罗奥克学院);Department of Civil Engineering(土木工程系);Department of Electronics and Communication(电子与通信系)
Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Oxford(牛津大学);Drexel University(德雷塞尔大学);Beijing University of Technology(北京理工大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Xi’an Jiaotong University(西安交通大学)
Anomalous Frame Detection by Grouping Frame Similarities between Two Videos Computed by Vision-Language Model to Extract Expert Workers' Unique Actions
Implicit Modeling for Transferability Estimation of Vision Foundation Models
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China(复杂与关键软件环境国家重点实验室,北京航空航天大学);School of Computer Science and Engineering, Beihang University, Beijing, China(计算机科学与工程学院,北京航空航天大学)
On Improving Adversarial Transferability of Vision Transformers
Australian National University(澳大利亚国立大学);Stony Brook University(石溪大学);Linköping University(林雪平大学);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Qualcomm USA(美国高通公司)
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Monash University(莫纳什大学);Temple University(天普大学);Peng Cheng Laboratory(鹏城实验室)
SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS(中国科学院香港创新研究院人工智能与机器人中心);Princeton University(普林斯顿大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
Towards Transferable Adversarial Attacks on Vision Transformers
Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University(复旦大学计算机科学技术学院上海市智能信息处理重点实验室);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心);Department of Computer Science, University of Maryland(马里兰大学计算机科学系)
Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models
National Institute of Health Data Science, Peking University(北京大学健康数据科学研究所);Institute of Medical Technology, Peking University(北京大学医学技术研究所);Deepwise Co., Ltd.(深度智医科技有限公司);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所)
Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features
School of Cyberspace Science and Technology, Beijing Institute of Technology(网络安全科学与技术学院,北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学)
Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement
Xi’an Jiaotong University(西安交通大学);Information Engineering University(信息工程大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Zhejiang Lab(之江实验室)
SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Sun Yat-sen University(中山大学)
Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
School of Cyber Science and Engineering, Southeast University(东南大学网络空间安全学院);Purple Mountain Laboratories(紫金山实验室);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
School of Computer Science, Fudan University(复旦大学计算机科学学院);Academy for Engineering & Technology, Fudan University(复旦大学工程技术学院);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai Engineering Research Center of AI & Robotics(上海人工智能与机器人工程研究中心);Engineering Research Center of AI & Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心)
Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
School of Artificial Intelligence and Computer Science, Jiangnan University(江南大学人工智能与计算机科学学院);Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(Surrey 大学视觉、语音和信号处理中心)