Despite widespread deployment of Large Language Models, systematic evaluation of instruction-following capabilities remains challenging. While comprehensive benchmarks exist, focused assessments that quickly diagnose specific instruction adherence patterns are valuable. As newer models may be trained on existing benchmarks, novel evaluation approaches are needed to assess genuine capabilities rather than memorized performance. This paper presents a streamlined evaluation framework using twenty carefully designed prompts to assess LLM instruction-following across diverse task categories. We demonstrate this framework through a large-scale empirical study conducted on October 14, 2025, testing 256 verified working models from 331 available via OpenRouter. To ensure methodological rigor and prevent selection bias, we first verified each model's basic functionality before inclusion. Unlike large-scale benchmarks requiring extensive computational resources, our approach offers a practical diagnostic tool researchers and practitioners can readily apply. Our methodology builds upon verifiable instructions while introducing a compact test suite balancing comprehensiveness with efficiency. Each prompt targets distinct aspects of instruction following, including format compliance, content constraints, logical sequencing, and multi-step task execution. We evaluate models from major providers (OpenAI, Anthropic, Google, Meta, Mistral) and emerging implementations (Qwen, DeepSeek, community models), providing comparative performance analysis. Our findings reveal consistent failure modes and identify specific instruction types posing particular challenges. This work contributes both a practical evaluation tool and one of the most comprehensive empirical analyses of instruction-following capabilities across the contemporary LLM landscape.
Offscript: Automated Auditing of Instruction Adherence in LLMs
University of Washington Information School Seattle Washington USA;University of Washington\ G. Allen School of Computer Science \& Engineering Seattle Washington USA;University of Washington Information School;University of Washington\ G. Allen School of Computer Science \& Engineering
Instructional Goal-Aligned Question Generation for Student Evaluation in Virtual Lab Settings: How Closely Do LLMs Actually Align?
Department of EdPsych, University of Wisconsin-Madison(威斯康星大学麦迪逊分校教育心理学系);Department of CS, Wichita State University(威斯康星州立大学Wichita分校计算机科学系);Department of CSSE, Auburn University(阿伯茨罕大学计算机科学与工程系)
Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models
The Chinese University of Hong Kong(香港中文大学);University of International Relations(国际关系大学);Tencent Jarvis Lab(腾讯Jarvis实验室);Westlake University(西湖大学);King’s College London(伦敦国王学院)
Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts
University of California, Los Angeles(加州大学洛杉矶分校);Sea AI Lab(Sea AI 实验室);Stanford University(斯坦福大学);University of Oxford(牛津大学);Yale University(耶鲁大学);NTU(南洋理工大学);NUS(新加坡国立大学);Boston University(波士顿大学)
RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format
Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理国家重点实验室(上海))
IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);IT University of Copenhagen(丹麦技术大学);Microsoft(微软)
MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following
The Pennsylvania State University(宾夕法尼亚州立大学);The Ohio State University(俄亥俄州立大学);Fudan University(复旦大学);Westlake University(西湖大学);Temple University(天普大学)
Role-aware Heuristic Episodic Attention for Conversational LLMs
National Key Laboratory of Parallel and Distributed Computing(并行与分布式计算国家重点实验室);College of Computer Science and Technology(计算机科学与技术学院);National University of Defense Technology(国防科技大学)
GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models
School of Information Sciences University of Illinois at Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校);Independent Researcher, Starc Institute(Starc研究所独立研究者);Computer Science and Engineering HKUST(HKUST计算机科学与工程学院);Computer Science Lapis Labs University of Illinois Urbana-Champaign(计算机科学Lapis Labs伊利诺伊大学厄巴纳-香槟分校);School of Information Sciences University of Illinois Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校)