Light-IF：通过预览和自检查赋予大语言模型通用推理能力以遵循复杂指令
Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
AI总结：
针对大语言模型遵循复杂指令时因惰性推理导致效果不佳的问题，提出Light-IF框架，通过预览和自检查机制及熵保持微调与强化学习，显著提升指令遵循能力，Light-IF-32B超越更大开源和闭源模型。
摘要：
尽管大语言模型推理能力的进步显著提升了其在解决数学问题、编程任务和一般谜题方面的表现，但它们在准确遵循指令方面的有效性仍不一致，尤其是在面对更复杂的指令时。我们的调查发现，思考阶段的惰性推理是导致指令遵循效果不佳的主要因素。为缓解这一问题，我们提出了一个综合框架，旨在实现涉及预览和自检查的严谨推理过程，这对于满足严格的指令约束至关重要。具体而言，我们首先生成带有复杂约束的指令，并应用过滤过程以获得有效提示，从而产生三个不同的提示数据集，分别归类为困难、简单和通过。然后，我们对通过提示采用拒绝采样来策划一个小而高质量的数据集，使模型能够进行冷启动初始化，并促进其适应有效的推理模式。随后，我们采用熵保持监督微调（Entropy-SFT）策略，结合基于规则的密集奖励引导的逐词熵自适应强化学习（TEA-RL）。这种方法鼓励模型转变其推理机制，最终培养涵盖预览和自检查的通用推理能力。在指令遵循基准上进行的大量实验表明，各种模型规模下均取得了显著的性能提升。值得注意的是，我们的Light-IF-32B模型超越了更大的开源模型（如DeepSeek-R1）和闭源模型（如豆包1.6）。
英文摘要：
While advancements in the reasoning abilities of LLMs have significantly enhanced their performance in solving mathematical problems, coding tasks, and general puzzles, their effectiveness in accurately adhering to instructions remains inconsistent, particularly with more complex directives. Our investigation identifies lazy reasoning during the thinking stage as the primary factor contributing to poor instruction adherence. To mitigate this issue, we propose a comprehensive framework designed to enable rigorous reasoning processes involving preview and self-checking, essential for satisfying strict instruction constraints. Specifically, we first generate instructions with complex constraints and apply a filtering process to obtain valid prompts, resulting in three distinct prompt datasets categorized as hard, easy, and pass. Then, we employ rejection sampling on the pass prompts to curate a small yet high-quality dataset, enabling a cold-start initialization of the model and facilitating its adaptation to effective reasoning patterns. Subsequently, we employ an entropy-preserving supervised fine-tuning (Entropy-SFT) strategy coupled with token-wise entropy-adaptive (TEA-RL) reinforcement learning guided by rule-based dense rewards. This approach encourages the model to transform its reasoning mechanism, ultimately fostering generalizable reasoning abilities that encompass preview and self-checking. Extensive experiments conducted on instruction-following benchmarks demonstrate remarkable performance improvements across various model scales. Notably, our Light-IF-32B model surpasses both larger open-source models such as DeepSeek-R1 and closed-source models like Doubao-1.6.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
