通过结构化表格决策模拟评估大语言模型（LLM）的理解能力
Evaluating LLM Understanding via Structured Tabular Decision Simulations
AI总结：
为评估LLM的真正理解能力，提出结构化表格决策模拟（STaDS）类专家决策评估套件，通过分析9个前沿LLM在15个场景的表现，发现多数模型跨域准确率不稳定且存在理由与决策因素不匹配问题，强调需全局评估协议及新型框架提升LLM理解能力。
摘要：
大语言模型（LLMs）通常能达到令人印象深刻的预测准确率，但仅靠正确性并不意味着真正的理解。真正的LLM理解能力，类似于人类专业知识，需要在多个实例和不同领域做出一致、有充分依据的决策，并依赖相关且基于领域的决策因素。我们引入结构化表格决策模拟（STaDS），这是一套类专家决策场景，将LLMs当作专业人员参加结构化决策“考试”来评估。在此语境下，理解被定义为识别并依赖正确决策因素的能力，这些因素是决定领域内结果的特征。STaDS通过以下方面联合评估理解能力：（i）问题与指令理解，（ii）基于知识的预测，（iii）对相关决策因素的依赖。通过分析9个前沿LLM在15个不同决策场景中的表现，我们发现：（a）大多数模型难以在不同领域持续获得高准确率；（b）模型可能准确率高但全局不可靠，且陈述的理由与驱动预测的因素之间常存在不匹配。我们的研究结果强调了对全局层面理解评估协议的需求，并倡导超越准确率的新型框架，以提升LLMs的理解能力。
英文摘要：
Large language models (LLMs) often achieve impressive predictive accuracy, yet correctness alone does not imply genuine understanding. True LLM understanding, analogous to human expertise, requires making consistent, well-founded decisions across multiple instances and diverse domains, relying on relevant and domain-grounded decision factors. We introduce Structured Tabular Decision Simulations (STaDS), a suite of expert-like decision settings that evaluate LLMs as if they were professionals undertaking structured decision ``exams''. In this context, understanding is defined as the ability to identify and rely on the correct decision factors, features that determine outcomes within a domain. STaDS jointly assesses understanding through: (i) question and instruction comprehension, (ii) knowledge-based prediction, and (iii) reliance on relevant decision factors. By analyzing 9 frontier LLMs across 15 diverse decision settings, we find that (a) most models struggle to achieve consistently strong accuracy across diverse domains; (b) models can be accurate yet globally unfaithful, and there are frequent mismatches between stated rationales and factors driving predictions. Our findings highlight the need for global-level understanding evaluation protocols and advocate for novel frameworks that go beyond accuracy to enhance LLMs' understanding ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
