arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.18892cs.CLcs.LG
收藏

当模型无法遵循指令时:测试256个大语言模型的指令遵循能力

When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs

University of Nevada Las Vegas(内华达大学拉斯维加斯分校)

Richard J. Young, Brandon Gillins, Alice M. Matthews

AI总结:

针对现有LLM指令遵循评估易受训练数据干扰的问题,提出含20个提示词的精简评估框架,测试256个主流及新兴LLM,揭示常见失效模式,提供实用诊断工具与全面实证分析。

摘要:

尽管大语言模型(LLM)已得到广泛部署,但对其指令遵循能力的系统性评估仍颇具挑战。虽然已有全面的基准测试,但能够快速诊断特定指令遵循模式的针对性评估仍具有重要价值。由于较新的模型可能是在现有基准上训练的,因此需要全新的评估方法来衡量其真实能力,而非记忆带来的表现。\n本文提出了一个精简的评估框架,使用20个精心设计的提示词,在不同任务类别下评估LLM的指令遵循能力。我们通过一项大规模实证研究验证了该框架,研究于2025年10月14日开展,测试了OpenRouter平台上331个可用模型中经确认可正常运行的256个模型。为确保方法严谨性并避免选择偏差,我们在纳入模型前先验证了每个模型的基本功能。与需要大量计算资源的大规模基准测试不同,我们的方法提供了一种研究人员和从业者可便捷使用的实用诊断工具。\n我们的方法以可验证的指令为基础,同时引入了一套兼顾全面性与效率的紧凑测试集。每个提示词都针对指令遵循的不同方面,包括格式合规性、内容约束、逻辑排序以及多步骤任务执行。我们评估了来自主流提供商(OpenAI、Anthropic、Google、Meta、Mistral)和新兴实现(Qwen、DeepSeek、社区模型)的模型,并提供了性能对比分析。研究结果揭示了一致的失效模式,并识别出具有特殊挑战性的特定指令类型。这项工作既贡献了一种实用的评估工具,也是对当前LLM领域指令遵循能力最全面的实证分析之一。

英文摘要:

Despite widespread deployment of Large Language Models, systematic evaluation of instruction-following capabilities remains challenging. While comprehensive benchmarks exist, focused assessments that quickly diagnose specific instruction adherence patterns are valuable. As newer models may be trained on existing benchmarks, novel evaluation approaches are needed to assess genuine capabilities rather than memorized performance. This paper presents a streamlined evaluation framework using twenty carefully designed prompts to assess LLM instruction-following across diverse task categories. We demonstrate this framework through a large-scale empirical study conducted on October 14, 2025, testing 256 verified working models from 331 available via OpenRouter. To ensure methodological rigor and prevent selection bias, we first verified each model's basic functionality before inclusion. Unlike large-scale benchmarks requiring extensive computational resources, our approach offers a practical diagnostic tool researchers and practitioners can readily apply. Our methodology builds upon verifiable instructions while introducing a compact test suite balancing comprehensiveness with efficiency. Each prompt targets distinct aspects of instruction following, including format compliance, content constraints, logical sequencing, and multi-step task execution. We evaluate models from major providers (OpenAI, Anthropic, Google, Meta, Mistral) and emerging implementations (Qwen, DeepSeek, community models), providing comparative performance analysis. Our findings reveal consistent failure modes and identify specific instruction types posing particular challenges. This work contributes both a practical evaluation tool and one of the most comprehensive empirical analyses of instruction-following capabilities across the contemporary LLM landscape.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑