辅助任务需求掩盖了较小语言模型的能力
Auxiliary task demands mask the capabilities of smaller language models
AI总结：
本文借鉴发展心理学“任务需求”概念，证明辅助任务需求会掩盖较小语言模型在推理等任务上的真实能力，强调模型表现受评估设计影响而非智力直接体现。
摘要：
发展心理学家一直争论语言理解或心智理论等认知能力何时出现。这些争论往往取决于“任务需求”的概念，即与执行特定评估相关的辅助挑战，这些挑战可能掩盖儿童潜在的能力。在测量语言模型（LMs）的能力时也会出现同样的问题：任务表现是模型潜在知识的函数，结合了模型在可用资源下解释和执行任务的能力。在此我们表明，对于类比推理、反思性推理、词汇预测和语法判断，任务需求较高的评估方法比需求较低的评估产生更低的性能。这种“需求差距”在参数较少和训练数据较少的模型中最为明显。我们的结果表明，LM的性能不应被解释为智力（或缺乏智力）的直接体现，而是通过研究人员设计选择的视角所反映出的能力。
英文摘要：
Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying knowledge, combined with the model's ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This "demand gap" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence (or lack thereof), but as a reflection of capacities seen through the lens of researchers' design choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
