AdaBoN: 适应性最佳N对齐
AdaBoN: Adaptive Best-of-N Alignment
AI总结：
本文提出一种适应性最佳N对齐策略，通过两阶段算法高效分配推理计算资源，实验证明其在不同提示批次中优于均匀分配方法。
摘要：
最近在测试时对齐方法上的进展，如最佳N采样，提供了一种简单有效的方法，通过奖励模型（RM）引导语言模型（LM）朝向期望行为。然而，这些方法可能计算成本高，尤其是在不考虑对齐难度差异的情况下统一应用。本文提出了一种提示适应策略，通过两阶段算法：初始探索阶段使用小探索预算估计每个提示的奖励分布，第二阶段根据这些估计自适应分配剩余预算。我们的方法简单、实用且兼容任何LM-RM组合。在AlpacaEval、HH-RLHF和PKU-SafeRLHF数据集上对12个LM/RM配对和50个不同提示批次的实验证明，我们的自适应策略在相同推理预算下优于均匀分配。此外，我们显示自适应策略在20%更大的推理预算下仍具有竞争力，并且随着批次大小增加，性能提升。
英文摘要：
Recent advances in test-time alignment methods, such as Best-of-N sampling, offer a simple and effective way to steer language models (LMs) toward preferred behaviors using reward models (RM). However, these approaches can be computationally expensive, especially when applied uniformly across prompts without accounting for differences in alignment difficulty. In this work, we propose a prompt-adaptive strategy for Best-of-N alignment that allocates inference-time compute more efficiently. Motivated by latency concerns, we develop a two-stage algorithm: an initial exploratory phase estimates the reward distribution for each prompt using a small exploration budget, and a second stage adaptively allocates the remaining budget using these estimates. Our method is simple, practical, and compatible with any LM-RM combination. Empirical results on prompts from the AlpacaEval, HH-RLHF, and PKU-SafeRLHF datasets for 12 LM/RM pairs and 50 different batches of prompts show that our adaptive strategy outperforms the uniform allocation with the same inference budget. Moreover, we show that our adaptive strategy remains competitive against uniform allocations with 20 percent larger inference budgets and improves in performance as the batch size grows.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
