arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14468cs.LG
收藏

LACONIC:面向LLM的长度感知约束强化学习

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

University of California, Los Angeles(加州大学洛杉矶分校);Salesforce AI Research(Salesforce AI研究院);National University of Singapore(新加坡国立大学);University of Alberta(阿尔伯塔大学)

Chang Liu, Yiran Zhao, Lawrence Liu, Yaoqi Ye, Csaba Szepesvári, Lin F. Yang

AI总结:

LACONIC通过长度感知约束强化学习,在保持任务性能的同时有效控制输出长度,减少50%以上。

摘要:

强化学习(RL)通过奖励驱动训练增强了大型语言模型(LLMs)的能力。然而,这一过程可能会产生过长的响应,增加推理延迟和计算开销。先前的长度控制方法通常依赖于固定的启发式奖励塑造,这可能与任务目标不一致,并且需要脆弱的调整。在本工作中,我们提出了LACONIC,一种在训练期间强制执行目标令牌预算的强化学习方法。具体来说,我们通过结合任务奖励与基于长度的成本的增强目标来更新策略模型。为了在简洁性和任务性能之间取得平衡,成本规模在整个训练过程中被动态调整。这产生了稳健的长度控制,同时保持任务奖励。我们提供了支持该方法的理论保证。在数学推理模型和数据集上,LACONIC保持或提高了pass@1的同时,将输出长度减少了超过50%。它在通用知识和多语言基准上保持了域外性能,使用44%更少的令牌。此外,LACONIC可以无缝集成到标准的RL训练中,无需推理更改和最小的部署开销。

英文摘要:

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑