STEMVerse: 一种用于大型语言模型中STEM推理的双轴诊断框架
STEMVerse: A Dual-Axis Diagnostic Framework for STEM Reasoning in Large Language Models
AI总结：
STEMVerse通过双轴诊断框架系统分析LLM在STEM领域的推理能力，揭示其结构失败模式，提供科学推理的深入理解。
摘要：
随着大型语言模型（LLMs）在复杂推理任务中取得显著突破，评估其在科学、技术、工程和数学（STEM）领域的专业能力已成为衡量机器智能的主要方法。然而，当前的评估范式往往将基准测试视为孤立的“孤岛”，仅提供单一的汇总分数，忽视了学术专精和认知深度的复杂性。这种以结果为导向的方法无法区分模型错误是源于领域知识不足还是认知能力缺陷，从而限制了诊断价值。为此，我们提出了STEMVerse，一种旨在系统分析LLMs STEM推理能力的诊断框架。该框架通过学术专精和认知复杂性两个维度来表征模型性能，以映射推理所需的能力。我们重新整合了超过20,000个来自主流基准测试的STEM问题，将其纳入统一的“学科×认知”能力空间，并为每个实例分配双轴标签。利用这一统一的诊断框架，我们系统地评估了不同参数规模和训练范式下的代表性LLM家族。我们的实证结果揭示了STEM推理中的结构失败模式。通过将多学科覆盖和细粒度认知分层整合到统一框架中，STEMVerse为理解LLMs的科学推理特性提供了清晰且可行的视角。
英文摘要：
As Large Language Models (LLMs) achieve significant breakthroughs in complex reasoning tasks, evaluating their proficiency in science, technology, engineering, and mathematics (STEM) has become a primary method for measuring machine intelligence. However, current evaluation paradigms often treat benchmarks as isolated "silos," offering only monolithic aggregate scores that neglect the intricacies of both academic specialization and cognitive depth. This result-oriented approach fails to distinguish whether model errors stem from insufficient domain knowledge or deficiencies in cognitive capacity, thereby limiting the diagnostic value. To address this, we propose STEMVerse, a diagnostic framework designed to systematically analyze the STEM reasoning capabilities of LLMs. This framework characterizes model performance across academic specialization and cognitive complexity to map the capability required for reasoning. We re-aggregate over 20,000 STEM problems from mainstream benchmarks into a unified "Discipline $\times$ Cognition" capability space, assigning dual-axis labels to every instance. Utilizing this unified diagnostic framework, we systematically evaluate representative LLM families across varying parameter scales and training paradigms. Our empirical results reveal structural failure patterns in STEM reasoning. By integrating multi-disciplinary coverage and fine-grained cognitive stratification into a unified framework, STEMVerse provides a clear and actionable perspective for understanding the scientific reasoning characteristics of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
