arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.08200cs.CVcs.LG
收藏

POP: Prompt Of Prompts for Continual Learning(面向持续学习的提示之提示)

POP: Prompt Of Prompts for Continual Learning

UC San Diego(加州大学圣地亚哥分校);Qualcomm AI Research(高通人工智能研究院);Qualcomm Technologies, Inc.(高通技术公司)

Zhiyuan Hu, Jiancheng Lyu, Dashan Gao, Nuno Vasconcelos

AI总结:

针对持续学习中特征空间语义漂移及任务专属与全局提示协同学习难题,提出POP模型,通过逐步学习任务专属提示与全局提示整合信息,在少样本设置下性能超越全量训练的经典方法。

摘要:

持续学习(Continual Learning,CL)近年来受到越来越多的关注,其目标是模拟人类在学习新概念时不发生灾难性遗忘的能力。现有持续学习方法虽在一定程度上实现了这一目标,但仍容易出现已学习特征空间的语义漂移问题。从超大规模数据集中习得、具备鲁棒特征表示能力的基础模型,为解决持续学习问题提供了理想的基础。近期研究也表明,可通过提示调优技术让基础模型适配特定任务,且该过程基本不会破坏特征表示的通用性。然而,一个尚未解决的问题是如何同时学习任务专属提示与全局提示(即捕捉跨任务信息的提示)。在本研究中,我们提出Prompt Of Prompts(POP,提示之提示)模型来解决这一问题:该模型逐步学习一组任务专属提示,以及一组被称为POP的全局提示,用以整合来自任务专属提示的信息。我们的实验表明,搭载POP学习机制的基础模型,性能能够大幅超越经典持续学习方法。此外,由于提示调优仅需少量训练样本,POP可在少样本设置下执行持续学习,且性能仍优于在完整数据集上训练的同类竞争方法。

英文摘要:

Continual learning (CL) has attracted increasing attention in the recent past. It aims to mimic the human ability to learn new concepts without catastrophic forgetting. While existing CL methods accomplish this to some extent, they are still prone to semantic drift of the learned feature space. Foundation models, which are endowed with a robust feature representation, learned from very large datasets, provide an interesting substrate for the solution of the CL problem. Recent work has also shown that they can be adapted to specific tasks by prompt tuning techniques that leave the generality of the representation mostly unscathed. An open question is, however, how to learn both prompts that are task specific and prompts that are global, i.e. capture cross-task information. In this work, we propose the Prompt Of Prompts (POP) model, which addresses this goal by progressively learning a group of task-specified prompts and a group of global prompts, denoted as POP, to integrate information from the former. We show that a foundation model equipped with POP learning is able to outperform classic CL methods by a significant margin. Moreover, as prompt tuning only requires a small set of training samples, POP is able to perform CL in the few-shot setting, while still outperforming competing methods trained on the entire dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑