PCoreSet：通过视觉-语言模型知识蒸馏实现的高效主动学习
PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
AI总结：
针对主动学习场景下缺乏任务特定教师模型的问题，提出融合视觉-语言模型知识蒸馏的ActiveKD框架及概率空间覆盖的PCoreSet选择策略，显著提升主动学习性能。
摘要：
知识蒸馏（KD）是一种广泛应用的框架，通过迁移教师模型的知识来训练紧凑的任务特定模型。然而，它在主动学习（AL）中的应用仍未得到充分探索，主动学习旨在通过迭代样本选择最大限度地降低标注成本。这一差距源于以下事实：知识蒸馏通常假设可以获取充足的标注数据，而主动学习运行在数据稀缺的场景中，这类场景下往往没有任务特定的教师模型可用。在本文中，我们首先提出ActiveKD框架，该框架利用大型视觉-语言模型（VLM）的零样本和少样本能力，将主动学习与知识蒸馏相结合。ActiveKD的一个关键特性是视觉-语言模型的结构化预测偏差——即它们的预测在概率空间中形成簇。我们将这种结构视为教师模型的归纳偏差，它捕捉了有利于学生模型学习的可泛化输出模式。为了利用这种偏差，我们提出了概率核心集（PCoreSet）选择策略，该策略最大化概率空间而非特征空间的覆盖范围。PCoreSet策略性地选择概率上多样化的未标注样本，在有限的标注预算下促进教师知识更高效的迁移。在11个数据集上的广泛评估表明，ActiveKD在各种选择方法中均能持续提升性能（例如在ImageNet上，各方法平均提升29.07%）。在ActiveKD框架下，PCoreSet在5个学生网络和3个教师网络的73个设置中有64个（约87.7%）排名第一，除了前2轮主动学习外，始终取得最佳性能。我们的代码可在https://github.com/erjui/PCoreSet获取。
英文摘要：
Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs-i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves performance across selection methods (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, PCoreSet ranks first in 64/73 settings (approximately 87.7%) across 5 student and 3 teacher networks, always achieving the best performance except for first 2 AL rounds. Our code is available at https://github.com/erjui/PCoreSet.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
