arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.07944cs.CV
收藏

训练空频视觉提示与概率簇以实现精准黑盒迁移学习

Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer Learning

Korea Advanced Institute of Science and Technology(韩国科学技术院);Letsur Inc.(Letsur公司)

Wonwoo Cho, Kangyeol Kim, Saemee Choi, Jaegul Choo

AI总结:

针对黑盒预训练模型迁移中的数据分布差异及资源受限问题,提出结合空频混合视觉提示与概率簇训练的参数高效迁移学习框架,在少样本视觉识别任务上性能优于现有基线,且能降低计算成本。

摘要:

尽管预测API服务等黑盒预训练模型(PTMs)的应用日益广泛,但由于数据分布差异,将通用模型直接应用于现实场景仍存在重大挑战。针对数据不足且计算资源受限的场景,本文提出了一种面向黑盒设置下视觉识别模型的新型参数高效迁移学习框架。该框架包含两种全新训练技术:其一,通过生成空间域与频率域的视觉提示,将预训练模型的输入空间(即图像)与目标数据分布对齐;其二,在提出空频混合视觉提示器的同时,设计了一种基于概率簇的新型训练技术,可增强输出空间(即预测概率)中的类别区分度。实验中,该模型在大量视觉识别数据集的少样本迁移学习设置下表现优异,性能超越了现有最优基线方法。此外,研究表明所提方法可有效降低训练与推理阶段的计算成本。

英文摘要:

Despite the growing prevalence of black-box pre-trained models (PTMs) such as prediction API services, there remains a significant challenge in directly applying general models to real-world scenarios due to the data distribution gap. Considering a data deficiency and constrained computational resource scenario, this paper proposes a novel parameter-efficient transfer learning framework for vision recognition models in the black-box setting. Our framework incorporates two novel training techniques. First, we align the input space (i.e., image) of PTMs to the target data distribution by generating visual prompts of spatial and frequency domain. Along with the novel spatial-frequency hybrid visual prompter, we design a novel training technique based on probabilistic clusters, which can enhance class separation in the output space (i.e., prediction probabilities). In experiments, our model demonstrates superior performance in a few-shot transfer learning setting across extensive visual recognition datasets, surpassing state-of-the-art baselines. Additionally, we show that the proposed method efficiently reduces computational costs for training and inference phases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑