arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.14688cs.LG
收藏

Q-Probe:一种面向语言模型奖励最大化的轻量级方法

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models

John A. Paulson School Of Engineering And Applied Sciences, Harvard University(哈佛大学约翰·保尔森工程与应用科学学院);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学坎普纳自然与人工智能研究所);Center of Mathematical Sciences and Applications, Harvard University(哈佛大学数学科学与应用中心)

Kenneth Li, Samy Jelassi, Hugh Zhang, Sham Kakade, Martin Wattenberg, David Brandfonbrener

AI总结:

提出 Q-probing 方法,通过在语言模型嵌入空间学习线性函数重加权候选补全以最大化任务奖励,介于微调与少样本提示之间,在数据受限时性能优于微调且仅需 API 访问权限。

摘要:

我们提出了一种名为 Q-probing 的方法,用于适配预训练语言模型以最大化特定任务的奖励函数。在宏观层面上,Q-probing 介于微调等较重方法和少样本提示等较轻方法之间,且可与两者结合。其核心思想是在模型的嵌入空间中学习一个简单的线性函数,用于对候选补全进行重新加权。我们从理论上证明,随着样本数量增加,该采样过程等价于对 Q-probe 的 KL 约束最大化。为了训练 Q-probes,我们考虑了奖励建模,或基于重要性加权策略梯度的一类新型直接策略学习目标。借助该技术,我们在具有真实奖励的领域(代码生成)以及由偏好数据定义的隐式奖励中均观察到性能提升,在数据受限的情况下甚至超越了微调。此外,Q-probe 可在 API 之上进行训练,因为它仅需访问采样和嵌入。代码:https://github.com/likenneth/q_probe 。

英文摘要:

We present an approach called Q-probing to adapt a pre-trained language model to maximize a task-specific reward function. At a high level, Q-probing sits between heavier approaches such as finetuning and lighter approaches such as few shot prompting, but can also be combined with either. The idea is to learn a simple linear function on a model's embedding space that can be used to reweight candidate completions. We theoretically show that this sampling procedure is equivalent to a KL-constrained maximization of the Q-probe as the number of samples increases. To train the Q-probes we consider either reward modeling or a class of novel direct policy learning objectives based on importance weighted policy gradients. With this technique, we see gains in domains with ground-truth rewards (code generation) as well as implicit rewards defined by preference data, even outperforming finetuning in data-limited regimes. Moreover, a Q-probe can be trained on top of an API since it only assumes access to sampling and embeddings. Code: https://github.com/likenneth/q_probe .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑