重播失败作为成功：用于指令跟随的样本高效强化学习
Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
AI总结：
HiR 提出了一种样本高效强化学习框架，通过将失败尝试视为成功来提升指令跟随任务的性能，利用双偏好学习实现高效优化。
摘要：
强化学习（RL）在对大型语言模型（LLMs）进行对齐以遵循指令方面展现出了潜力，能够处理各种约束条件。尽管结果令人鼓舞，但RL的进步不可避免地依赖于采样成功的高质量响应；然而，初始模型往往难以生成满足所有约束条件的响应，由于其能力有限，导致稀疏或无法区分的奖励，阻碍了学习。在本工作中，我们提出了一种名为 hindsight instruction replay（HiR）的新样本高效 RL 框架，用于复杂的指令跟随任务。该框架采用选择然后重写策略，基于事后已满足的约束条件将失败尝试视为成功。我们在这些重播样本以及原始样本上进行 RL，理论上将目标框架为双偏好学习，以在指令和响应层面实现高效的优化，仅使用二元奖励信号。广泛的实验表明，所提出的 HiR 在不同的指令跟随任务中取得了有希望的结果，同时需要较少的计算预算。我们的代码和数据集可在 https://github.com/sastpg/HIR 上获得。
英文摘要：
Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and dataset is available at https://github.com/sastpg/HIR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
