重新审视指令跟随中语言模型的可靠性
Revisiting the Reliability of Language Models in Instruction-Following
AI总结：
本文提出可靠@k指标和自动生成相似提示的流水线，构建IFEval++基准，发现当前模型在细微差异提示下性能下降高达61.8%，并探索了三种改进方法。
摘要：
先进的LLM在IFEval等基准测试中已达到接近上限的指令跟随准确率。然而，这些令人印象深刻的分数并不一定能转化为实际使用中的可靠服务，因为用户经常改变他们的措辞、上下文框架和任务表述。在本文中，我们研究面向细微差异的可靠性：模型是否在传达类似用户意图但具有细微差异的相似提示中表现出一致的能力。为了量化这一点，我们引入了一个新的指标，可靠@k，并开发了一个自动化流水线，通过数据增强生成高质量的相似提示。在此基础上，我们构建了IFEval++用于系统评估。在20个专有和26个开源LLM中，我们发现当前模型在面向细微差异的可靠性方面存在显著不足——它们的性能在细微提示修改下可能下降高达61.8%。此外，我们对其进行了表征，并探索了三种潜在的改进方法。我们的发现强调了面向细微差异的可靠性是朝着更可靠和可信的LLM行为迈出的关键但尚未充分探索的下一步。我们的代码和基准可访问：https://github.com/jianshuod/IFEval-pp。
英文摘要：
Advanced LLMs have achieved near-ceiling instruction-following accuracy on benchmarks such as IFEval. However, these impressive scores do not necessarily translate to reliable services in real-world use, where users often vary their phrasing, contextual framing, and task formulations. In this paper, we study nuance-oriented reliability: whether models exhibit consistent competence across cousin prompts that convey analogous user intents but with subtle nuances. To quantify this, we introduce a new metric, reliable@k, and develop an automated pipeline that generates high-quality cousin prompts via data augmentation. Building upon this, we construct IFEval++ for systematic evaluation. Across 20 proprietary and 26 open-source LLMs, we find that current models exhibit substantial insufficiency in nuance-oriented reliability -- their performance can drop by up to 61.8% with nuanced prompt modifications. What's more, we characterize it and explore three potential improvement recipes. Our findings highlight nuance-oriented reliability as a crucial yet underexplored next step toward more dependable and trustworthy LLM behavior. Our code and benchmark are accessible: https://github.com/jianshuod/IFEval-pp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
