arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.17067cs.CLcs.AI
收藏

评估大型语言模型中的模糊任务

Evaluating Ill-Defined Tasks in Large Language Models

IBM Research San Jose, CA(IBM研发San Jose分校)

Yi Zhou, Basel Shbita

AI总结:

本文分析了现有评估基准在处理模糊任务时的不足,通过复杂指令跟随和自然语言到Mermaid序列图的案例研究,揭示了评估方法的不稳定性与非诊断性,推动更稳健的评估设计。

摘要:

许多大型语言模型(LLM)的评估针对本质上模糊的任务,这些任务具有不明确的输入输出空间和模糊的成功标准。我们分析现有评估基准和指标为何无法可靠或诊断性地反映此类任务模型的能力。我们研究了两个案例:复杂指令跟随(CIF),其中识别出重复问题包括现实世界指令复杂度覆盖有限、对指令措辞敏感、指标不一致且不可比、以及由基于LLM的评委引入的不稳定性;以及自然语言到Mermaid序列图(NL2Mermaid),其中展示了多维评估标准如何提供超越聚合分数的可行见解。这些案例研究表明,当前评估经常混淆不同的失败模式,导致分数不稳定、非诊断且难以采取行动。我们的发现揭示了现有评估实践中对模糊任务的根本限制,并推动了更稳健、可解释的评估设计。

英文摘要:

Many evaluations of Large Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why existing evaluation benchmarks and metrics fail to provide reliable or diagnostic signals of model capability for such tasks. We examine two case studies: Complex Instruction Following (CIF), where we identify recurring issues including limited coverage of real-world instruction complexity, sensitivity to instruction phrasing, inconsistent and non-comparable metrics, and instability introduced by LLM-based judges; and Natural Language to Mermaid Sequence Diagrams (NL2Mermaid), where we show how multi-faceted evaluation criteria can yield actionable insights beyond aggregate scores. Together, these case studies show that current evaluations frequently conflate distinct failure modes, yielding scores that are unstable, non-diagnostic, and difficult to act upon. Our findings expose fundamental limitations in existing evaluation practices for ill-defined tasks and motivate more robust, interpretable evaluation designs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑