arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.06868cs.CLcs.AI
收藏

SwitchPrompt:学习特定于领域的门控软阈值,用于低资源领域的分类

SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains

Adobe Research(奥多比研究院);Bosch Center for Artificial Intelligence(博世人工智能中心);Bosch Research North America(博世北美研究院)

Koustava Goswami, Lukas Lange, Jun Araki, Heike Adel

摘要:

对预训练的语言模型进行分类会在自然语言处理任务中产生有希望的结果,但由于预训练数据和下游任务之间的域间隙,当应用于低资源域时效果较差。在这项工作中,我们弥合这一差距与一种新的和轻量级的提示方法称为SwitchPrompt的适应语言模型训练的数据集从一般域到不同的低资源域。SwitchPrompt使用领域特定的关键字和可训练的门控提示,提供面向领域的提示,即对通用领域语言模型的目标领域的有效指导。我们在三个文本分类基准上的Few-Shot实验证明了与SwitchPrompt一起使用时通用域预训练语言模型的有效性。他们的表现甚至超过了使用最先进的基线提示方法训练的特定领域的同行,准确率提高了10.7%。这个结果表明SwitchPrompt有效地减少了对特定于领域的语言模型预训练的需求。

英文摘要:

Prompting pre-trained language models leads to promising results across natural language processing tasks but is less effective when applied in low-resource domains, due to the domain gap between the pre-training data and the downstream task. In this work, we bridge this gap with a novel and lightweight prompting methodology called SwitchPrompt for the adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Using domain-specific keywords with a trainable gated prompt, SwitchPrompt offers domain-oriented prompting, that is, effective guidance on the target domains for general-domain language models. Our few-shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompting methods by up to 10.7% performance increase in accuracy. This result indicates that SwitchPrompt effectively reduces the need for domain-specific language model pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑