通过双头优化从视觉-语言模型进行简单而有效的半监督知识蒸馏
Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
AI总结：
针对半监督知识蒸馏中监督与蒸馏损失的梯度冲突，本文提出 DHO 双头优化，以极小开销稳定提升特征学习，并在 15 个数据集及 ImageNet 设置中达到最先进性能。
摘要：
半监督学习（SSL）已成为通过利用无标注数据来应对数据稀缺挑战的实用解决方案。近来，在海量图文对上预训练的视觉-语言模型（VLM）展现出显著的零样本/少样本性能；凭借其出色的泛化能力，这些性能往往超过 SSL 方法。这一差距促使我们追问：如何才能有效地将 VLM 强大的泛化能力引入任务特定模型？知识蒸馏（KD）为迁移 VLM 能力提供了自然框架，但我们发现它存在监督损失与蒸馏损失之间的梯度冲突。为应对这一挑战，我们提出双头优化（Dual-Head Optimization，DHO），为每个不同信号引入双预测头。我们观察到，DHO 解决了梯度冲突，相较于单头 KD 基线能够实现更好的特征学习，并具有计算开销极小以及无需重新训练即可在测试时调节超参数的实际优势。在 15 个数据集上的大量实验表明，DHO 始终优于 KD 基线，并且常常用更小的学生模型超过教师模型。DHO 还在分布内的 ImageNet 半监督学习以及跨 ImageNet 变体的分布外泛化两方面取得了新的最先进性能。我们公开发布代码和模型检查点，以促进未来研究，地址为 https://github.com/erjui/DHO。
英文摘要：
Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Recently, vision-language models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance that often surpasses SSL approaches due to their exceptional generalization capabilities. This gap motivates us to question: how can we effectively harness the powerful generalization capabilities of VLMs into task-specific models? Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but we identify that it suffers from gradient conflicts between supervised and distillation losses. To address this challenge, we propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal. We observe that DHO resolves gradient conflicts, enabling improved feature learning compared to single-head KD baselines, with practical benefits of minimal computational overhead and test-time hyperparameter tuning without retraining. Extensive experiments across 15 datasets show that DHO consistently outperforms KD baselines, often outperforming teacher models with smaller student models. DHO also achieves new state-of-the-art performance on both in-distribution ImageNet semi-supervised learning and out-of-distribution generalization across ImageNet variants. We publicly release our code and model checkpoints to facilitate future research at https://github.com/erjui/DHO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
