ALoRE:通过聚合低秩专家实现高效视觉适配
ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts
AI总结:
针对现有单分支参数高效迁移学习方法解耦能力不足的问题,提出多分支聚合低秩专家的ALoRE方法,仅用极少参数就在24项图像分类任务上性能超越全微调和其他主流PETL方法。
摘要:
参数高效迁移学习(PETL)已成为将大规模视觉基础模型适配到下游任务的一种极具前景的范式。典型方法主要利用内在低秩特性进行分解,在压缩参数量的同时学习任务特定权重。然而,这类方法大多采用单分支结构在原始特征空间内进行操作,这对于解耦所学表征和模式而言可能并非最优。本文提出了一种新型PETL方法ALoRE,该方法复用由克罗内克积构建的超复参数化空间,采用多分支范式聚合低秩专家(Low Rank Experts),在训练过程中解耦所学的认知模式。得益于这一巧妙设计,ALoRE仅增加可忽略不计的额外参数,并且可通过重参数化以顺序方式轻松合并到冻结的主干网络中,避免了额外的推理延迟。我们使用多种主干网络变体在24项图像分类任务上开展了大量实验。实验结果表明,ALoRE在性能和参数效率方面均优于全微调策略及其他最先进的PETL方法。例如,ALoRE仅更新0.15M参数,在FGVC数据集和VTAB-1k基准上的平均Top-1准确率相比全微调分别提升了3.06%和9.97%。
英文摘要:
Parameter-efficient transfer learning (PETL) has become a promising paradigm for adapting large-scale vision foundation models to downstream tasks. Typical methods primarily leverage the intrinsic low rank property to make decomposition, learning task-specific weights while compressing parameter size. However, such approaches predominantly manipulate within the original feature space utilizing a single-branch structure, which might be suboptimal for decoupling the learned representations and patterns. In this paper, we propose ALoRE, a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts using a multi-branch paradigm, disentangling the learned cognitive patterns during training. Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone via re-parameterization in a sequential manner, avoiding additional inference latency. We conduct extensive experiments on 24 image classification tasks using various backbone variants. Experimental results demonstrate that ALoRE outperforms the full fine-tuning strategy and other state-of-the-art PETL methods in terms of performance and parameter efficiency. For instance, ALoRE obtains 3.06% and 9.97% Top-1 accuracy improvement on average compared to full fine-tuning on the FGVC datasets and VTAB-1k benchmark by only updating 0.15M parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。