arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.11537cs.CVcs.LG
收藏

语义提示与图像令牌用于持续学习

Semantic Prompting with Image-Token for Continual Learning

Ajou University(亚洲大学);Tech. Innovation Group, KT(KT技术创新集团)

Jisu Han, Jaemin Na, Wonjun Hwang

AI总结:

针对持续学习中任务选择不完善导致性能下降的问题,提出基于图像令牌语义的I-Prompt方法,通过语义匹配与令牌级提示消除任务预测,在四个基准上实现竞争性能并显著减少训练时间。

摘要:

持续学习旨在为新任务优化模型参数,同时保留来自先前任务的知识。近期,基于提示的学习方法应运而生,利用预训练模型通过提示来学习后续任务,而无需依赖回放缓冲区。尽管这种方法已展现出卓越的结果,现有方法仍依赖于先前的任务选择过程来挑选合适的提示。然而,任务选择的不完善可能会对性能产生负面影响,尤其是在任务数量众多或任务分布不均衡的场景中。为解决此问题,我们引入了I-Prompt,一种与任务无关的方法,专注于图像令牌的视觉语义信息以消除任务预测。我们的方法包括语义提示匹配,即基于令牌之间的相似性来确定提示,以及图像令牌级提示,即将提示直接应用于中间层的图像令牌。因此,我们的方法在四个基准上取得了具有竞争力的性能,同时与最先进的方法相比显著减少了训练时间。此外,我们通过大量实验展示了我们的方法在各种场景下的优越性。

英文摘要:

Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently, prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results, existing methods depend on preceding task-selection process to choose appropriate prompts. However, imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic approach focuses on the visual semantic information of image tokens to eliminate task prediction. Our method consists of semantic prompt matching, which determines prompts based on similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens in the intermediate layers. Consequently, our method achieves competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Moreover, we demonstrate the superiority of our method across various scenarios through extensive experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑