Many evaluations of Large Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why existing evaluation benchmarks and metrics fail to provide reliable or diagnostic signals of model capability for such tasks. We examine two case studies: Complex Instruction Following (CIF), where we identify recurring issues including limited coverage of real-world instruction complexity, sensitivity to instruction phrasing, inconsistent and non-comparable metrics, and instability introduced by LLM-based judges; and Natural Language to Mermaid Sequence Diagrams (NL2Mermaid), where we show how multi-faceted evaluation criteria can yield actionable insights beyond aggregate scores. Together, these case studies show that current evaluations frequently conflate distinct failure modes, yielding scores that are unstable, non-diagnostic, and difficult to act upon. Our findings expose fundamental limitations in existing evaluation practices for ill-defined tasks and motivate more robust, interpretable evaluation designs.
The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
Iris AI Borgen, Norway(Iris AI 布尔根,挪威);Iris AI Neurobiology BAS Sofia, Bulgaria(Iris AI 神经生物学 BAS 萨福ia,保加利亚);Iris AI Sofia, Bulgaria(Iris AI 萨福ia,保加利亚)
VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室)