AI 中文总结
研究围绕多方法框架构建CogArena基准评估大语言模型认知能力结构，通过多模型实验，发现范式相关性多为正但组内优势不明显，冻结确认标准及复制均失败，未建立稳定五维分布，提供了结合多种要素的工作流程。
AI 中文摘要
大语言模型（LLM）的认知分数越来越多地被总结为按能力分布，其维度应在不同任务中趋同，对匹配干预有选择性反应，并能推广到用于定义它们的模型之外。我们引入了CogArena，这是一个围绕多方法框架构建的程序生成的13范式基准，用于确定认知任务分数何时在五个理论驱动的分组中获得维度标签。在55个开放权重模型中，几乎所有范式相关性都是正的，一个共同轴解释了约一半的方差。组内优势小、对分数敏感且在模型家族间不确定。在对六个家族的12个模型进行的单独冻结、完全交叉研究中，有针对性的支架显示出小的匹配组优势，但没有支架特异性对比在多重校正后幸存，选择性也没有改善对未参与家族的预测。冻结确认标准失败。事后的替代措辞复制产生了较小的正估计且再次失败。这些结果共同支持一个边界结论。理论对齐的提示产生了小的电池内对角线趋势，但目前的证据并未建立稳定的五维分布。CogArena提供了一个在将认知标签附加到模型分数之前，结合行为特征、协方差、匹配干预和家族外预测的工作流程。
英文摘要
LLM cognitive scores are increasingly summarized as per-ability profiles whose dimensions should converge across tasks, respond selectively to matched interventions, and generalize beyond the models used to define them. We introduce CogArena, a procedurally generated 13-paradigm benchmark built around a multimethod framework for determining when cognitive-task scores warrant dimensional labels across five theory-motivated groupings. Across 55 open-weight models, nearly all paradigm correlations are positive and a common axis explains about half the variance. The within-grouping advantage is small, scoring-sensitive, and uncertain across model families. In a separately frozen, fully crossed study across 12 models from six families, targeted scaffolds show a small matched-grouping advantage, but no scaffold-specific contrast survives multiplicity correction and selectivity does not improve held-out-family prediction. The frozen confirmation criterion fails. A post-hoc alternate-wording replication produces a smaller positive estimate and again fails. Together, these results support a boundary conclusion. Theory-aligned prompting produces a small in-battery diagonal tendency, but the present evidence does not establish stable five-dimensional profiles. CogArena provides a workflow joining behavioral signatures, covariance, matched interventions, and out-of-family prediction before cognitive labels are attached to model scores.
Comments21 pages, 8 figures. Code and the procedurally generated battery: https://github.com/dengzhe-hou/CogArena
