SPaR：利用带树搜索精炼的自我博弈提升大语言模型的指令遵循能力
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models
AI总结：
本文提出自我博弈框架 SPaR，通过树搜索自我精炼构建低干扰、可比较的偏好对，提升 LLM 指令遵循能力，并在 IFEval 上验证其效果与可扩展性。
摘要：
指令遵循是语言模型的一项基础能力，要求模型识别指令中即使最细微的要求，并将其准确反映在输出中。这种能力非常适合通过偏好学习进行优化，而且通常也正是如此优化的。然而，现有方法在构建偏好对时，往往直接从模型采样多个相互独立的回复。这种做法会引入与指令是否被精确遵循无关的内容变化（例如，对相同语义的不同表达），从而干扰教导模型识别那些能够改善指令遵循的关键差异这一目标。鉴于此，我们提出 SPaR，这是一个整合树搜索自我精炼的自我博弈框架，用于生成有效、可比较且不受干扰的偏好对。通过与自身博弈，LLM 采用树搜索策略，根据指令精炼其先前回复，同时尽量减少不必要的变化。实验表明，在 SPaR 指导下经过三轮迭代训练的 LLaMA3-8B 模型，在 IFEval 基准上超过 GPT-4-Turbo，且未损失通用能力。此外，SPaR 展现出良好的可扩展性，显著增强了 GLM-4-9B 和 LLaMA3-70B 等模型。我们还分析了树搜索中的推理扩展如何影响模型性能。代码和数据已公开发布于 https://github.com/thu-coai/SPaR。
英文摘要：
Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output. Such an ability is well-suited for and often optimized by preference learning. However, existing methods often directly sample multiple independent responses from the model when creating preference pairs. Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following. In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations. Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. Furthermore, SPaR demonstrates promising scalability, greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how inference scaling in tree search would impact model performance. Our code and data are publicly available at https://github.com/thu-coai/SPaR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
