大规模因子分析表明机器智能仅部分可解释
Large-scale factor analysis shows machine intelligence is only partially interpretable
- KAIST（韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究通过大规模因子分析发现，语言模型中的通用智能因子仅部分可解释，且未被标准基准良好代理，挑战了当前将通用智能作为具体构念的开发策略。
AI中文摘要：
在语言模型开发中，一个常见的假设是认知能力围绕一个通用的、领域无关的智能因子组织，类似于人类的流体智力。这一假设很少被直接检验，且先前的尝试仅在更小的规模上进行。我们采用潜变量方法研究语言模型中的智能，类似于心理测量学家研究心理构念的方式。每个特定问题集的表现受领域特定和领域无关的潜因子影响。使用因子分析作为降维技术，我们分析了涵盖1,618个语言模型在456个不同纯文本基准上的13,251个已发表评估分数。由于数据集的超稀疏性质，我们通过不同的数据稠化方法和插补方法对分析进行三角验证。在不同偏差模式下，一个稳健的模式是：1. 在我们最慷慨的估计中，一个通用智能因子解释了模型表现方差的70.8%，而在大多数解决方案中远低于此；2. 内容相似的基准不一定聚集在一起；3. g因子不受任何共同主题主导，且缺乏证据表明它被标准的“智能”基准很好地代理。我们的发现与当前在语言模型开发中将通用智能定义、识别和定位为具体构念的努力相悖。这使得针对单一概念能力的策略缺乏支持，因为其必须达到的一阶能力通常是部分特异的，在实践中无法识别。
英文摘要：
A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard "intelligence" benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.
