arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06279cs.CL
收藏

多提示解码器有助于更好的语言理解

Multi-Prompting Decoder Helps Better Language Understanding

National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学)

Zifeng Cheng, Zhaoling Chen, Zhiwei Jiang, Yafeng Yin, Cong Wang, Shiping Ge, Qing Gu

AI总结:

针对模型即服务设置下单一提示质量依赖问题,提出多提示解码器(MPD)框架,通过多提示查询和最优传输、校准解码策略,在少样本自然语言理解任务上取得最优结果。

摘要:

近期的预训练语言模型(PLMs)通常仅向用户提供推理API,即新兴的模型即服务设置。为了在不访问参数和梯度的情况下将MaaS PLMs适配到下游任务,一些现有方法侧重于PLMs的输出端适配,将PLM视为编码器,随后优化特定任务的解码器,用于解码PLM的输出隐藏状态和类别分数。尽管这些方法有效,但它们仅使用单一提示查询PLMs进行解码,导致严重依赖所采用提示的质量。在本文中,,我们提出了一个简单有效的多提示解码器框架用于MaaS适配。核心思想是用多个不同提示对每个样本查询PLMs,从而获得多个输出隐藏状态和类别分数以供后续解码。这种多提示解码范式能同时减轻对单一提示质量的依赖,缓解少样本设置下的数据稀缺问题,并提供从PLMs中提取的更丰富知识。具体而言,我们提出两种解码策略:针对隐藏状态使用最优传输的多提示解码,以及针对类别分数的校准解码。大量实验表明,我们的方法在少样本设置下的多个自然语言理解数据集上取得了新的最优结果。

英文摘要:

Recent Pre-trained Language Models (PLMs) usually only provide users with the inference APIs, namely the emerging Model-as-a-Service (MaaS) setting. To adapt MaaS PLMs to downstream tasks without accessing their parameters and gradients, some existing methods focus on the output-side adaptation of PLMs, viewing the PLM as an encoder and then optimizing a task-specific decoder for decoding the output hidden states and class scores of the PLM. Despite the effectiveness of these methods, they only use a single prompt to query PLMs for decoding, leading to a heavy reliance on the quality of the adopted prompt. In this paper, we propose a simple yet effective Multi-Prompting Decoder (MPD) framework for MaaS adaptation. The core idea is to query PLMs with multiple different prompts for each sample, thereby obtaining multiple output hidden states and class scores for subsequent decoding. Such multi-prompting decoding paradigm can simultaneously mitigate reliance on the quality of a single prompt, alleviate the issue of data scarcity under the few-shot setting, and provide richer knowledge extracted from PLMs. Specifically, we propose two decoding strategies: multi-prompting decoding with optimal transport for hidden states and calibrated decoding for class scores. Extensive experiments demonstrate that our method achieves new state-of-the-art results on multiple natural language understanding datasets under the few-shot setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑