arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.13028cs.CL
收藏

基于强化学习的上下文学习在Incomplete Utterance Rewriting中的应用

In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

Haowei Du, Dongyan Zhao

AI总结:

针对ICL中示例选择未利用LLM反馈的问题,提出基于强化学习的RLS框架,通过LM选择器和LLM生成器优化示例选择,显著提升了少样本设置下的性能。

摘要:

大型语言模型(LLMs)的上下文学习(ICL)在社区中引起了越来越多的关注,其中LLMs仅基于增加了少量示例的指令进行预测。现有的ICL示例选择方法利用稀疏或密集检索器,并获得了有效的性能。然而,这些方法没有利用LLM的直接反馈来训练检索器,且选择的示例不一定能提高LLM的类比能力。为了解决这个问题,我们提出了一种基于策略的强化学习示例选择框架(RLS),它由一个语言模型(LM)选择器和一个LLM生成器组成。LM选择器将候选示例编码为密集表示,并选择前k个示例作为LLM的演示。LLM的输出被用来计算奖励和策略梯度,以优化LM选择器。我们在不同的数据集上进行了实验,并显著优于现有的示例选择方法。此外,我们的方法在少样本设置中显示出优于监督微调(SFT)模型的优势。进一步的实验表明,示例的丰富度以及与测试用例的相似性之间的平衡对于LLM的ICL性能至关重要。

英文摘要:

In-context learning (ICL) of large language models (LLMs) has attracted increasing attention in the community where LLMs make predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. However, these methods do not utilize direct feedback of LLM to train the retriever and the examples selected can not necessarily improve the analogy ability of LLM. To tackle this, we propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator. The LM selector encodes the candidate examples into dense representations and selects the top-k examples into the demonstration for LLM. The outputs of LLM are adopted to compute the reward and policy gradient to optimize the LM selector. We conduct experiments on different datasets and significantly outperform existing example selection methods. Moreover, our approach shows advantages over supervised finetuning (SFT) models in few shot setting. Further experiments show the balance of abundance and the similarity with the test case of examples is important for ICL performance of LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑