几何感知蒸馏用于提示调优生物医学视觉-语言模型
Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models
AI总结：
提出Omni-Geometry知识蒸馏（OGKD）框架，通过注入类别关系结构到教师模型，生成保留真实标签同时尊重类间几何的方向性目标，并设计全局几何感知蒸馏（GAD）和标签引导几何蒸馏（LGD）损失，在11个医学数据集上平均提升准确率1.7%-2.8%。
摘要：
当前基于提示和适配器的视觉-语言模型（VLM）调优方法在医学影像中具有吸引力，因为临床数据敏感性倾向于冻结骨干网络且标注有限。然而，这些方法通常仅优化真实类别，将所有其他类别视为同等错误，忽略了临床上有意义的类别关系，并在有限监督设置下产生不稳定的决策边界。我们提出了Omni-Geometry知识蒸馏（OGKD），一种新框架，将类别关系结构注入教师模型，以生成保留真实标签同时尊重类间几何的方向性目标。利用这些目标，我们开发了两种蒸馏损失：全局几何感知蒸馏（GAD）作用于全局图像标记，标签引导几何蒸馏（LGD）将相同的几何应用于注意力补丁标记以改善细粒度对齐。在11个广泛使用的医学数据集上进行的基础到新类和少样本评估的综合实验和分析中，我们的OGKD实现了显著更好的性能，在所有先前最先进的VLM适应方法上平均绝对增益为1.7%-2.8%。它还能稳健地泛化到未见类别，并产生比其他方法更可靠的预测。我们的代码可在https://github.com/tientrandinh/OGKD获取。
英文摘要：
Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging, where clinical data sensitivity favors frozen backbones and annotations are limited. However, these methods typically optimize only the ground-truth class, treating all other classes as equally incorrect, ignoring clinically meaningful class relations and yielding unstable decision boundaries in limited-supervision settings. We propose Omni-Geometry Knowledge Distillation (OGKD), a new framework that injects class-relation structure into the teacher to produce directional targets that preserve the ground truth while respecting inter-class geometry. Using these targets, we develop two distillation losses: Global Geometry-Aware Distillation (GAD) operates on the global image token, and Label-Guided Geometry Distillation (LGD) applies the same geometry to attentive patch tokens to improve fine-grained alignment. Across comprehensive experiments and analyses on 11 widely-used medical datasets for base-to-novel and few-shot evaluations, our OGKD achieves substantially better performance, consistently improving accuracy by an average absolute gain of 1.7%-2.8% over all prior state-of-the-art VLM adaptation counterparts. It also robustly generalizes to unseen classes and yields more reliable predictions than other approaches. Our code is available at https://github.com/tientrandinh/OGKD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
