arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.06208cs.CL
收藏

面向以人为中心的文本理解的个性化LoRA

Personalized LoRA for Human-Centered Text Understanding

Yunnan University(云南大学);Yuan Ze University(元智大学)

You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, Xuejie Zhang

AI总结:

针对以人为中心的文本理解中预训练语言模型适配难的问题,提出带即插即用框架的个性化LoRA,结合个性化丢弃和互信息最大化策略,在更少参数下于多场景下优于现有方法。

摘要:

在以人为中心的文本理解(HCTU)任务中,有效且高效地适配预训练语言模型(PLM)颇具挑战,因为在大多数个性化应用中用户token达百万级,且不具备具体的显式语义。标准的参数高效方法(如LoRA)需要为每个用户存储大量适配器组。本文提出一种带有即插即用(PnP)框架的个性化LoRA(PLoRA)用于HCTU任务,PLoRA效果优异、参数高效且可在PLM中动态部署。此外,该方法采用个性化丢弃策略和互信息最大化策略,因此能很好地适配少样本/零样本学习场景,解决冷启动问题。在四个基准数据集上的实验表明,尽管可训练参数更少,该方法在HCTU任务的全样本/少样本/零样本场景下均优于现有方法。为保证可复现性,论文代码已发布于:https://github.com/yoyo-yun/PLoRA。

英文摘要:

Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑