arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.20795cs.CL
收藏

通过文本模型预测控制对大型语言模型进行测试时对齐

Test-Time Alignment for Large Language Models via Textual Model Predictive Control

National Yang Ming Chiao Tung University;NVIDIA;National Taiwan University

Kuang-Da Wang, Teng-Ruei Chen, Yu Heng Hung, Guo-Xun Ko, Shuoyang Ding, Yueh-Hua Wu, Yu-Chiang Frank Wang, Chao-Han Huck Yang, Wen-Chih Peng, Ping-Chun Hsieh

AI总结:

本文提出TMPC方法,通过文本模型预测控制解决测试时对齐问题,通过分层强化学习原理改进生成过程,提升不同任务的性能。

摘要:

对齐大型语言模型(LLMs)与人类偏好通过微调是资源密集型的,推动了测试时轻量级替代方案的发展。本文通过序列决策制定的视角,揭示了两个根本性挑战。当动作在令牌层面定义时,如引导解码,对齐会受到地平线诅咒的影响。相反,当动作在响应层面定义时,如传统迭代细化,会遇到维度诅咒。为解决这一权衡,本文受到控制理论中模型预测控制(MPC)的启发,提出文本模型预测控制(TMPC),一种适用于推理时对齐LLMs的新型预测规划框架。标准MPC的限制在于其依赖于预定义的、硬性段边界,这在文本生成中通常不存在。TMPC通过引入受分层强化学习启发的两个原则来克服这一问题:(1)回顾性子目标识别,其中TMPC分析生成子目标以回顾性地识别高奖励的中间输出作为子目标。这使得框架能够发现有意义的、任务特定的规划步骤(例如机器翻译中的句子或代码生成中的bug fix)。(2)子目标条件重生成,其中这些识别出的子目标用于指导后续的规划迭代。通过在这些已证明的高质量子目标上进行条件,TMPC通过在之前已验证的成功基础上进行改进,确保了稳定性。TMPC在三个具有不同分段属性的任务上进行了评估:话语层面翻译、长形式响应生成和程序合成。结果表明,TMPC在不同任务上都提高了性能,突显了其通用性。

英文摘要:

Aligning Large Language Models (LLMs) with human preferences through finetuning is resource-intensive, motivating lightweight alternatives at test time. We address test-time alignment through the lens of sequential decision making, a perspective that reveals two fundamental challenges. When actions are defined at the token level, as in guided decoding, alignment suffers from the curse of horizon. Conversely, when actions are at the response level, as in traditional iterative refinement, the curse of dimensionality emerges. To resolve this trade-off, we draw inspiration from Model Predictive Control (MPC) in control theory to propose Textual Model Predictive Control (TMPC), a novel predictive planning framework adapted for aligning LLMs at inference time. A key limitation of standard MPC is its reliance on predefined, hard segment boundaries, which are often absent in text generation. TMPC overcomes this by introducing two principles inspired by hierarchical reinforcement learning: (1) Hindsight Subgoal Identification, where TMPC analyzes generation subgoals to retrospectively identify high-reward intermediate outputs as subgoals. This allows the framework to discover meaningful, task-specific planning steps (e.g., a sentence in machine translation or a bug fix in code generation.). (2) Subgoal-Conditioned Re-Generation, where these identified subgoals are used to guide subsequent planning iterations. By conditioning on these proven, high-quality subgoals, TMPC ensures stable improvement by building upon previously validated successes. TMPC is evaluated on three tasks with distinct segmentation properties: discourse-level translation, long-form response generation, and program synthesis. The results demonstrate that TMPC consistently improves performance, highlighting the generality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑