可验证指令遵循的泛化
Generalizing Verifiable Instruction Following
AI总结：
针对模型在精准指令遵循任务中对基准约束过拟合、泛化能力弱的问题，本文提出IFBench新基准评估58个新域外约束的泛化性，设计约束验证模块并通过RLVR提升性能，还发布相关训练资源。
摘要：
成功的人机与AI交互的关键因素之一是语言模型或聊天机器人精准遵循人类指令的能力。指令的常见特征是用户为生成更有用答案而添加的输出约束，如“仅用是或否回答”或“至少提及‘abrakadabra’一词3次”。即使是当今最强大的模型也难以满足此类约束。我们发现，大多数模型在测试这些能力的基准中，对一小部分可验证约束存在严重过拟合（该能力称为精准指令遵循），无法很好地泛化到未见过的输出约束。我们引入新基准IFBench，用于在58个新的、多样且具挑战性的域外可验证约束上评估精准指令遵循的泛化能力。此外，我们对模型如何以及在何种数据上训练以提升精准指令遵循泛化能力进行了广泛分析：精心设计约束验证模块，并表明基于可验证奖励的强化学习（RLVR）能显著改善指令遵循。除IFBench外，我们还发布29个额外的人工标注训练约束、验证函数、RLVR训练提示词及代码。
英文摘要：
A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
