HeBA：异构瓶颈适配器用于鲁棒的视觉-语言模型
HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models
AI总结：
HeBA通过引入模态特定的结构归纳偏置，改进了视觉-语言模型在下游任务中的适应性，提升了稳定性和准确性，达到11个少样本基准的新状态。
摘要：
适应大规模视觉-语言模型（VLMs）如CLIP到下游任务时，通常面临“一刀切”架构问题，其中视觉和文本标记被统一处理。我们提出HeBA（异构瓶颈适配器），一种统一的架构框架，引入模态特定的结构归纳偏置。HeBA通过三个关键架构创新：（1）异质性：使用2D深度可分离卷积处理视觉标记以保留空间相关性，使用密集线性投影处理文本标记以捕捉语义关系；（2）瓶颈正则化：不同于标准扩展适配器，HeBA采用压缩瓶颈（D->D/4）强制模型学习紧凑、鲁棒的特征并作为结构正则化器；（3）主动梯度初始化：挑战限制性的零初始化范式，利用Kaiming初始化策略确保足够的初始梯度流以加速收敛而不影响冻结主干的预训练知识。大量实验表明，HeBA的架构专业化设计实现了更高的稳定性和准确性，在11个少样本基准上建立了新的状态。代码可在https://github.com/Jahid12012021/VLM-HeBA获取。
英文摘要：
Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a "one-size-fits-all" architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D -> D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
