Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. Our code is available at: https://github.com/hey-cjj/MoVE-KD.
Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization
Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系）；Center of Machine Intelligence and Data Science (C-MInDS), IIT Bombay（印度理工学院班加罗尔人工智能与数据科学中心）
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Department of Information Technology（信息科技系）；Electrical Engineering, ETH Zurich, Zurich, Switzerland（电气工程，苏黎世联邦理工学院，苏黎世，瑞士）；Qualcomm AI Research, Amsterdam, the Netherlands（高通人工智能研究，阿姆斯特丹，荷兰）；Department of Electrical, Electronic and Information Engineering（电气、电子与信息工程系）；University of Bologna, Bologna, Italy（博洛尼亚大学，博洛尼亚，意大利）；School of Electrical and Electronic Engineering（电气与电子工程学院）
DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
Department of Computer Science（计算机科学系）；Engineering, Ohio State University, Ohio, US（工程系，俄亥俄州立大学，俄亥俄，美国）；Department of Computer Science, University of Central Florida, Florida, US（计算机科学系，中央佛罗里达大学，佛罗里达，美国）
Adversarial Prompt Distillation for Vision-Language Models
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理实验室，计算机科学学院，复旦大学）；The Chinese University of Hong Kong, Shatin, Hong Kong（香港中文大学，沙田，香港）；The University of Hong Kong, Pokfulam, Hong Kong（香港大学，薄扶林，香港）
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（信息科技学院，科学大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Bridging Weakly-Supervised Learning and VLM Distillation: Noisy Partial Label Learning for Efficient Downstream Adaptation
Institute of Perceptual Intelligence, Peng Cheng Laboratory（感知智能研究院，鹏城实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, China（通用人工智能国家重点实验室，智能科学与技术学院，北京大学，中国）；Huawei Noah’s Ark Lab, China（华为诺亚方舟实验室，中国）
FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation
School of Artificial Intelligence, Beijing University of Posts；School of National Elite Institute of Engineering, Chongqing University, Chongqing, China；The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China；The Key Laboratory of Computing Power Network；Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology, Jinan, China.；Shandong Provincial Key Laboratory of Computing Power Internet；Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, China. -1cm
Tianjin University（天津大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sheffield（谢菲尔德大学）
Vision Language Models (VLMs), pre-trained on large-scale image-text datasets, enable zero-shot predictions for unseen data but may underperform on specific unseen tasks. Continual learning (CL) can help VLMs effectively adapt to new data distributions without joint training, but faces challenges of catastrophic forgetting and generalization forgetting. Although significant progress has been achieved by distillation-based methods, they exhibit two severe limitations. One is the popularly adopted single-teacher paradigm fails to impart comprehensive knowledge, The other is the existing methods …
Dual-Model Distillation for Efficient Action Classification with Hybrid Edge-Cloud Solution
University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Saratoga High School（萨拉托加高中）；Ridge High School（里奇高中）；The Harker School（哈克学校）；University of California Santa Cruz（加利福尼亚大学圣克鲁兹分校）
Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving
ECE Department（电子工程系）；Carnegie Mellon University（卡内基梅隆大学）；Computer Science Department（计算机科学系）；Columbia University（哥伦比亚大学）；Rotman School of Management（罗特曼管理学院）；University of Toronto（多伦多大学）；Department of Statistics（统计学系）；George Washington University（乔治华盛顿大学）；Department of Computer Science（计算机科学系）；San Francisco State University（旧金山州立大学）
Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation
College of Intelligent Systems Science and Engineering, Harbin Engineering University（哈尔滨工程大学智能系统科学与工程学院）；Northwest Institute of Mechanical and Electrical Engineering（西北机电工程研究所）
Pedestrian crossing is one of the most typical behavior which conflicts with natural driving behavior of vehicles. Consequently, pedestrian crossing prediction is one of the primary task that influences the vehicle planning for safe driving. However, current methods that rely on the practically collected data in real driving scenes cannot depict and cover all kinds of scene condition in real traffic world. To this end, we formulate a deep virtual to real distillation framework by introducing the synthetic data that can be generated conveniently, and borrow the abundant information of pedestria…
