arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.12895cs.CL
收藏

测试时偏好优化:通过迭代文本反馈实现即时对齐

Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback

Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)

Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, Yu Cheng

AI总结:

提出测试时偏好优化(TPO)框架,利用文本反馈在推理阶段迭代优化LLM输出,无需重训练即可对齐人类偏好,实验显示TPO能提升对齐效果并高效扩展。

摘要:

大型语言模型(LLM)表现出令人印象深刻的性能,但缺乏无需重新训练即可快速适应人类偏好的灵活性。在这项工作中,我们引入了测试时偏好优化(TPO),这是一个在推理过程中将LLM输出与人类偏好对齐的框架,无需更新模型参数。TPO不依赖纯数值奖励,而是将奖励信号转化为文本批评,并将其作为文本奖励来迭代优化其响应。在涵盖指令遵循、偏好对齐、安全性和数学的基准测试上的评估表明,TPO逐步提高了与人类偏好的对齐程度。值得注意的是,仅经过几步TPO后,最初未对齐的Llama-3.1-70B-SFT模型就能超越对齐的对应模型Llama-3.1-70B-Instruct。此外,TPO在推理过程中随搜索宽度和深度高效扩展。通过案例研究,我们展示了TPO如何利用LLM解释和响应奖励信号的内在能力。我们的研究结果确立了TPO作为测试时偏好优化的实用、轻量级替代方案,实现即时对齐。我们的代码公开于https://github.com/yafuly/TPO。

英文摘要:

Large language models (LLMs) demonstrate impressive performance but lack the flexibility to adapt to human preferences quickly without retraining. In this work, we introduce Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference, removing the need to update model parameters. Rather than relying on purely numerical rewards, TPO translates reward signals into textual critiques and uses them as textual rewards to iteratively refine its response. Evaluations on benchmarks covering instruction following, preference alignment, safety, and mathematics reveal that TPO progressively improves alignment with human preferences. Notably, after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO scales efficiently with both the search width and depth during inference. Through case studies, we illustrate how TPO exploits the innate capacity of LLM to interpret and act upon reward signals. Our findings establish TPO as a practical, lightweight alternative for test-time preference optimization, achieving alignment on the fly. Our code is publicly available at https://github.com/yafuly/TPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑