激励推理以提升大语言模型的高级指令遵循能力
Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
AI总结：
针对大语言模型复杂指令遵循能力弱、普通思维链效果不佳的问题，提出RAIF方法，通过可验证奖励强化学习和专家行为克隆激励推理，使1.5B模型性能提升11.74%，媲美8B模型且泛化性良好。
摘要：
现有大语言模型（LLMs）在遵循复杂指令方面面临挑战，尤其是当存在多个约束且这些约束以并行、链式和分支结构组织时。一种直观的解决方案即思维链（CoT）被认为能普遍提升LLMs的能力。然而我们发现，普通CoT由于仅对指令进行释义的表层推理模式，会对性能产生负面影响，它无法拆解约束的构成，以识别其在类型和维度层级间的关系。为此，我们提出RAIF，这是一种通过在测试时扩展计算量来激励推理、提升LLMs处理复杂指令能力的系统性方法。首先，我们基于现有分类体系对复杂指令进行分解，提出了一种可复现的数据获取方法。其次，我们利用带有可验证规则中心奖励信号的强化学习（RL），专门培养指令遵循方面的推理能力。我们通过逐样本对比来强化优质CoT，解决复杂指令下推理的浅层、非本质问题。我们还利用专家行为克隆，推动从“快思考”LLMs到熟练推理者的平稳分布偏移。在7个综合基准上的大量评估验证了所提方法的有效性：一个1.5B参数的LLM获得了11.74%的性能提升，表现与8B参数的LLM相当。在分布外（OOD）约束上的评估也证实了RAIF的泛化性。代码和数据可在https://github.com/yuleiqin/RAIF获取。
英文摘要：
Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF. Codes and data are available at https://github.com/yuleiqin/RAIF. Keywords: reinforcement learning with verifiable rewards (RLVR), instruction following, complex instructions
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
