在线上下文蒸馏用于低资源视觉语言模型
Online In-Context Distillation for Low-Resource Vision Language Models
AI总结：
本文提出在线上下文蒸馏方法，通过小模型与强教师模型协作，利用稀疏演示高效缩小性能差距，提升小模型性能达33%，在低资源条件下实现与零样本性能相当的成果。
摘要：
随着领域对更多资源的追求，本文聚焦于一个关键问题：如何使视觉语言模型（VLMs）在低资源、预算受限的环境下适应？尽管大型VLMs表现强劲，但在这种环境下不切实际。小VLMs效率高，但通常需要昂贵的微调来缩小与大模型在部署领域性能差距。受上下文学习框架启发，我们提出在线上下文蒸馏（ICD）方法，其中小VLM在推理时与更强的教师模型协作，通过稀疏演示蒸馏其知识，高效弥合两者差距。我们的方法基于深入分析，识别了当前视觉语言ICL可行的模型规模和选择，并展示了在受限计算预算下ICL优于微调的优势。我们通过新颖的跨模态演示选择策略、教师测试时缩放以减少噪声，以及学生不确定性条件来动态填充演示池并最小化教师查询。我们的ICD方法显著提升了小模型性能（最高提升33%），使用稀缺教师注释（最低4%），并在零样本性能上竞争。
英文摘要：
As the field continues its push for ever more resources, this work turns the spotlight on a critical question: how can vision-language models (VLMs) be adapted to thrive in low-resource, budget-constrained settings? While large VLMs offer strong performance, they are impractical to deploy in such settings. Small VLMs, on the other hand, are efficient but typically require costly fine-tuning to close the performance gap with larger models in the deployment domain. Inspired by the in-context learning framework, we propose an online In-Context Distillation (ICD) method, in which a small VLM collaborates with a stronger teacher model at inference time, distilling its knowledge via sparse demonstrations to efficiently bridge the gap between them. Our method is built on an in-depth analysis that identifies the scale and the choice of models for which vision-language ICL is currently feasible, and demonstrates the advantage of ICL over fine-tuning under constrained compute budgets. We enhance our method with a novel cross-modal demonstration selection strategy, teacher test-time scaling to reduce noise, and student uncertainty conditioning to dynamically populate a demonstration pool and minimize teacher queries. Our ICD method significantly boosts the performance of small models (up to 33%) using scarce teacher annotations (as low as 4%), and competes with the teacher's zero-shot performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
