arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15713cs.CLcs.AI
收藏

让预训练语言模型成为更好的持续少样本关系抽取器

Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors

Beijing University of Posts and Telecommunications(北京邮电大学)

Shengkun Ma, Jiale Han, Yi Liang, Bo Cheng

AI总结:

本文提出对比提示学习框架和记忆增强策略,利用预训练语言模型的隐式能力,有效缓解持续少样本关系抽取中的灾难性遗忘和过拟合问题,并在低资源场景下显著超越现有方法。

摘要:

持续少样本关系抽取(CFRE)是一个实际问题,要求模型在只有少量标注训练数据的情况下持续学习新关系,同时避免遗忘旧关系。主要挑战是灾难性遗忘和过拟合。本文利用提示学习来探索预训练语言模型的隐式能力,以应对上述两个挑战,从而使语言模型成为更好的持续少样本关系抽取器。具体来说,我们提出了一个对比提示学习框架,该框架设计提示表示以获取更具泛化性的知识,使其能够容易地适应旧类别和新类别,并采用基于间隔的对比学习来更加关注困难样本,从而缓解灾难性遗忘和过拟合问题。为了进一步改善低资源场景下的过拟合问题,我们引入了一种有效的记忆增强策略,利用精心设计的提示来引导ChatGPT生成多样化的样本。大量实验表明,我们的方法大幅优于最先进的方法,并显著缓解了低资源场景下的灾难性遗忘和过拟合。

英文摘要:

Continual Few-shot Relation Extraction (CFRE) is a practical problem that requires the model to continuously learn novel relations while avoiding forgetting old ones with few labeled training data. The primary challenges are catastrophic forgetting and overfitting. This paper harnesses prompt learning to explore the implicit capabilities of pre-trained language models to address the above two challenges, thereby making language models better continual few-shot relation extractors. Specifically, we propose a Contrastive Prompt Learning framework, which designs prompt representation to acquire more generalized knowledge that can be easily adapted to old and new categories, and margin-based contrastive learning to focus more on hard samples, therefore alleviating catastrophic forgetting and overfitting issues. To further remedy overfitting in low-resource scenarios, we introduce an effective memory augmentation strategy that employs well-crafted prompts to guide ChatGPT in generating diverse samples. Extensive experiments demonstrate that our method outperforms state-of-the-art methods by a large margin and significantly mitigates catastrophic forgetting and overfitting in low-resource scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑