基于潜在层次适配器学习的细粒度视觉语言模型微调
Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
AI总结：
针对现有适配器无法捕捉类别与图像间一对多关联及未知类别对齐难题，提出潜在层次适配器 LatHAdapter，通过可学习属性提示和双曲空间层次正则化学习潜在语义层次，在少样本分类任务中显著优于现有微调方法。
摘要：
基于适配器的方法在少样本分类任务中对预训练视觉语言模型（Vision-Language Models，VLMs）进行微调已引起广泛关注。这些方法致力于开发一个轻量级模块，以更好地对齐视觉表示与（类别）文本表示，从而提升下游少样本学习任务的性能。然而，现有适配器通常通过底层嵌入空间中的显式空间邻近性来学习/对齐（类别）文本-视觉模态，这 i) 无法捕捉类别与图像样本之间固有的一对多关联，并且 ii) 难以在未知类别与图像之间建立准确的关联。为了解决这些问题，受近期双曲学习研究的启发，我们提出了一种新颖的潜在层次适配器（Latent Hierarchical Adapter，LatHAdapter），用于在下游少样本分类任务上微调视觉语言模型。LatHAdapter 的核心在于利用下游训练数据的潜在语义层次结构，并利用其为适配器学习过程提供更丰富、更细粒度的指导。具体而言，LatHAdapter 首先引入一些可学习的“属性”提示作为连接类别与图像的桥梁。然后，它将每个批次中的类别、属性提示和图像投影到双曲空间中，并采用层次正则化来学习它们的潜在语义层次结构，从而充分建模类别、可学习属性与图像样本之间固有的一对多关联。在四个具有挑战性的少样本任务上的大量实验表明，所提出的 LatHAdapter 始终优于许多其他微调方法，尤其是在适应已知类别和泛化到未知类别方面。
英文摘要：
Adapter-based approaches have garnered attention for fine-tuning pre-trained Vision-Language Models (VLMs) on few-shot classification tasks. These methods strive to develop a lightweight module that better aligns visual and (category) textual representations, thereby enhancing performance on downstream few-shot learning tasks. However, existing adapters generally learn/align (category) textual-visual modalities via explicit spatial proximity in the underlying embedding space, which i) fails to capture the inherent one-to-many associations between categories and image samples and ii) struggles to establish accurate associations between the unknown categories and images. To address these issues, inspired by recent works on hyperbolic learning, we develop a novel Latent Hierarchical Adapter (LatHAdapter) for fine-tuning VLMs on downstream few-shot classification tasks. The core of LatHAdapter is to exploit the latent semantic hierarchy of downstream training data and employ it to provide richer, fine-grained guidance for the adapter learning process. Specifically, LatHAdapter first introduces some learnable `attribute' prompts as the bridge to align categories and images. Then, it projects the categories, attribute prompts, and images within each batch in a hyperbolic space, and employs hierarchical regularization to learn the latent semantic hierarchy of them, thereby fully modeling the inherent one-to-many associations among categories, learnable attributes, and image samples. Extensive experiments on four challenging few-shot tasks show that the proposed LatHAdapter consistently outperforms many other fine-tuning approaches, particularly in adapting known classes and generalizing to unknown classes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
