arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.04320cs.CL
收藏

用于上下文学习的统一示例检索器

Unified Demonstration Retriever for In-Context Learning

Fudan University(复旦大学);East China Normal University(华东师范大学);Pingan Health Tech(平安健康科技)

Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, Xipeng Qiu

AI总结:

提出统一示例检索器UDR,通过多任务列表式排序训练框架与迭代挖掘策略,利用语言模型反馈整合多任务信号,在30多个任务上显著优于基线,并展现出跨模型规模与场景的强大泛化能力。

摘要:

上下文学习是一种新的学习范式,语言模型根据少量输入-输出对(示例)和测试输入进行条件化,并直接输出预测结果。研究表明,该范式高度依赖于所提供的示例,因此推动了示例检索的研究:给定测试输入,从训练集中检索相关示例,作为上下文学习的信息性示例。尽管先前的研究专注于为多个任务分别训练任务特定的检索器,但这些方法通常难以在各种任务间迁移和扩展,且单独训练的检索器会产生大量的参数存储和部署成本。在本文中,我们提出了统一示例检索器(Unified Demonstration Retriever, UDR),这是一种用于为广泛任务检索示例的单一模型。为了训练UDR,我们利用语言模型的反馈,将各种任务的训练信号转化为统一的列表式排序公式。随后,我们提出了一种多任务列表式排序训练框架,并配以迭代挖掘策略来寻找高质量候选示例,这有助于UDR充分整合各种任务的信号。在13个任务族和多个数据领域的30多个任务上的实验表明,UDR显著优于基线方法。进一步分析表明,所提出的每个组件均有效,且UDR在各种场景下均表现出强大的能力,包括不同的语言模型(1.3B - 175B)、未见过的数据集、不同的示例数量等。

英文摘要:

In-context learning is a new learning paradigm where a language model conditions on a few input-output pairs (demonstrations) and a test input, and directly outputs the prediction. It has been shown highly dependent on the provided demonstrations and thus promotes the research of demonstration retrieval: given a test input, relevant examples are retrieved from the training set to serve as informative demonstrations for in-context learning. While previous works focus on training task-specific retrievers for several tasks separately, these methods are often hard to transfer and scale on various tasks, and separately trained retrievers incur a lot of parameter storage and deployment cost. In this paper, we propose Unified Demonstration Retriever (\textbf{UDR}), a single model to retrieve demonstrations for a wide range of tasks. To train UDR, we cast various tasks' training signals into a unified list-wise ranking formulation by language model's feedback. Then we propose a multi-task list-wise ranking training framework, with an iterative mining strategy to find high-quality candidates, which can help UDR fully incorporate various tasks' signals. Experiments on 30+ tasks across 13 task families and multiple data domains show that UDR significantly outperforms baselines. Further analyses show the effectiveness of each proposed component and UDR's strong ability in various scenarios including different LMs (1.3B - 175B), unseen datasets, varying demonstration quantities, etc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑