LLaVA-CKD: 从下至上级联知识蒸馏用于视觉-语言模型
LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
AI总结：
本文提出LLaVA-CKD框架，通过级联知识蒸馏提升视觉-语言模型效率，通过中间容量教师模型逐步提升学生模型性能，并在七个VQA基准上验证了其最优性能。
摘要：
大型视觉-语言模型（VLMs）在解决视觉-语言理解任务如视觉问答（VQA）方面表现出色，但其内存和计算需求仍对实际部署构成挑战。知识蒸馏是一种有前景的技术，通过将高容量教师网络的知识转移到较小的学生网络中。然而，两者之间的容量差距既是福音也是诅咒：学生网络越小，效率越高，而教师网络越大，携带的知识越多；但超过一定点后，两者之间的容量差距越大，知识转移效果越差。为应对这一影响，我们提出了一种从下至上的级联知识蒸馏（CKD）框架。受人类正式教育体系的启发，我们引入一个（可能多个）中间容量的教师模型，逐步将学生网络提升到下一个级别，其中下一个（更高容量）的教师可以接管。我们提供了一种理论分析，以研究级联蒸馏对学生成绩的影响。我们将在基于LLaVA方法构建的模型上应用该框架，并在七个标准、公开可用的VQA基准上评估衍生模型，证明了其最优性能。
英文摘要：
Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question Answering (VQA), but their memory and compute requirements remain a concern for practical deployment. A promising class of techniques for mitigating this concern is Knowledge Distillation, where knowledge from a high-capacity Teacher network is transferred to a considerably smaller Student network. However, the capacity gap between the two networks is both a blessing and a curse: the smaller the Student network, the better its efficiency, and the larger the Teacher, the more knowledge it carries; yet, beyond a point, the larger capacity gap between the two leads to worse knowledge transfer. To counter this effect, we propose a bottom-up cascaded knowledge distillation (CKD) framework. Instead of treating knowledge transfer as an activity involving one high-capacity Teacher (or an ensemble of such), inspired by human formal education systems, we introduce one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over. We provide a theoretical analysis in order to study the effect of cascaded distillation in the generalization performance of the Student. We apply the proposed framework on models build upon the LLaVA methodology and evaluate the derived models on seven standard, publicly available VQA benchmarks, demonstrating their SotA performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
