NAS-LoRA: 通过可搜索适应增强视觉基础模型的参数高效微调
NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
AI总结：
NAS-LoRA 通过引入可搜索适应的神经网络架构搜索块，提升视觉基础模型在特定领域中的参数高效微调性能，减少训练成本24.14%。
摘要：
Segment Anything Model (SAM) 已经成为图像分割领域强大的视觉基础模型。然而，将 SAM 调整到特定下游任务，如医学和农业成像，仍然是一项重大挑战。为此，低秩适应（LoRA）及其变种已被广泛用于增强 SAM 在不同领域上的适应性能。尽管有进展，一个关键问题出现了：我们能否将归纳偏置整合到模型中？这尤其相关，因为 SAM 中的 Transformer 编码器本质上缺乏图像块内的空间先验，可能阻碍高阶语义信息的获取。在本文中，我们提出 NAS-LoRA，一种新的参数高效微调（PEFT）方法，旨在弥合预训练 SAM 与专用领域之间的语义差距。具体而言，NAS-LoRA 在 LoRA 的编码器和解码器组件之间加入了一个轻量级的神经网络架构搜索（NAS）块，以动态优化整合到权重更新中的先验知识。此外，我们提出了一种分阶段优化策略，帮助 ViT 编码器平衡权重更新和架构调整，从而逐步学习高阶语义信息。各种实验表明，我们的 NAS-LoRA 提高了现有 PEFT 方法的性能，同时通过减少 24.14% 的训练成本，而无需增加推理成本，突显了 NAS 在增强视觉基础模型 PEFT 方面的潜力。
英文摘要：
The Segment Anything Model (SAM) has emerged as a powerful visual foundation model for image segmentation. However, adapting SAM to specific downstream tasks, such as medical and agricultural imaging, remains a significant challenge. To address this, Low-Rank Adaptation (LoRA) and its variants have been widely employed to enhancing SAM's adaptation performance on diverse domains. Despite advancements, a critical question arises: can we integrate inductive bias into the model? This is particularly relevant since the Transformer encoder in SAM inherently lacks spatial priors within image patches, potentially hindering the acquisition of high-level semantic information. In this paper, we propose NAS-LoRA, a new Parameter-Efficient Fine-Tuning (PEFT) method designed to bridge the semantic gap between pre-trained SAM and specialized domains. Specifically, NAS-LoRA incorporates a lightweight Neural Architecture Search (NAS) block between the encoder and decoder components of LoRA to dynamically optimize the prior knowledge integrated into weight updates. Furthermore, we propose a stage-wise optimization strategy to help the ViT encoder balance weight updates and architectural adjustments, facilitating the gradual learning of high-level semantic information. Various Experiments demonstrate our NAS-LoRA improves existing PEFT methods, while reducing training cost by 24.14% without increasing inference cost, highlighting the potential of NAS in enhancing PEFT for visual foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
