arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.20194cs.CL
收藏

GAPO:通过生成对抗策略优化学习偏好提示

GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization

Fudan University(复旦大学);Alibaba Group(阿里巴巴集团)

Zhouhong Gu, Xingzhou Chen, Xiaoran Shi, Tao Wang, Suhang Zheng, Tianyu Li, Hongwei Feng, Yanghua Xiao

AI总结:

GAPO提出结合GAN训练动态与仅编码器奖励模型的框架,通过对抗生成难度渐增样本,逐步学习复杂约束,在细粒度约束场景中优于PPO、DPO和KTO,实现更稳健的LLM输出控制。

摘要:

近期大语言模型的进展凸显了通过预定义约束对模型输出进行精确控制的迫切需求。现有方法试图通过直接指令-响应合成或偏好响应优化来实现这一目标,但往往在约束理解和适应方面存在困难。在处理细粒度约束时,这一局限性尤为明显,导致产生幻觉或性能脆弱。我们提出了生成对抗策略优化(GAPO),这是一种新颖的框架,将基于GAN的训练动态与仅编码器奖励模型相结合,逐步学习并适应日益复杂的约束。GAPO利用对抗训练自动生成难度各异的训练样本,同时利用仅编码器架构更好地捕捉提示-响应关系。大量实验表明,GAPO在多个基准上表现出优越性能,尤其是在需要细粒度约束处理的场景中,显著优于PPO、DPO和KTO等现有方法。我们的结果表明,GAPO在偏好提示学习上的独特方法为控制LLM输出提供了一种更稳健、更有效的解决方案。代码可在https://github.com/MikeGu721/GAPO获取。

英文摘要:

Recent advances in large language models have highlighted the critical need for precise control over model outputs through predefined constraints. While existing methods attempt to achieve this through either direct instruction-response synthesis or preferential response optimization, they often struggle with constraint understanding and adaptation. This limitation becomes particularly evident when handling fine-grained constraints, leading to either hallucination or brittle performance. We introduce Generative Adversarial Policy Optimization (GAPO), a novel framework that combines GAN-based training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. GAPO leverages adversarial training to automatically generate training samples of varying difficulty while utilizing the encoder-only architecture to better capture prompt-response relationships. Extensive experiments demonstrate GAPO's superior performance across multiple benchmarks, particularly in scenarios requiring fine-grained constraint handling, where it significantly outperforms existing methods like PPO, DPO, and KTO. Our results suggest that GAPO's unique approach to preferential prompt learning offers a more robust and effective solution for controlling LLM outputs. Code is avaliable in https://github.com/MikeGu721/GAPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑