arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.00992cs.AIcs.CLcs.CYcs.HC
收藏

大型语言模型在规则应用中展现出概念掌握能力的证据

Evidence of conceptual mastery in the application of rules by Large Language Models

Pontifical Catholic University of Rio de Janeiro(里约热内卢天主教大学);INSPER(因斯佩尔学院);Maynooth University(梅努斯大学)

José Luiz Nunes, Guilherme FCF Almeida, Brian Flanagan

AI总结:

本文通过两项心理学实验发现,Gemini Pro等部分LLMs在规则应用上展现类人表现,证实LLMs掌握规则概念,对法律决策与哲学探究有启示。

摘要:

本文利用心理学方法研究大型语言模型(LLMs)在规则应用方面的概念掌握能力,我们引入一种新程序,将LLMs产生的思维多样性与人类样本的思维多样性进行匹配。随后开展两项实验,对比人类与LLMs基于规则的决策情况:研究1发现,所有被研究的LLMs都复制了人类的模式,无论这些模型是用训练截止日期之前还是之后生成的场景进行提示;此外,我们发现人类在两组场景间存在未预料到的差异,令人惊讶的是,LLMs的响应也复制了这些差异。研究2聚焦人类规则应用的语境特征:在强制延迟条件下,人类样本更依赖规则的文本,而非规则目的等其他考量;我们的结果显示,部分模型(Gemini Pro和Claude 3)对描述强制延迟或时间压力的提示做出类人反应,而其他模型(GPT-4o和Llama 3.2 90b)则未表现出该特性。我们认为,收集到的证据表明LLMs掌握了规则概念,这对法律决策和哲学探究均具有启示意义。

英文摘要:

Background. Evidence that large language models (LLMs) reproduce human judgments does not establish conceptual mastery: the correspondence may reflect memorisation or be sensitivite to incidental task features. Objective. Across five experiments, we test whether 13 LLMs possess a generalisable competence in applying rules, including cases in which a rule's text and purpose point towards different outcomes. Method. Study 1A compared LLM judgments with newly collected human data on published stimuli and matched vignettes created after the models' training cut-offs. Studies 2A/2B compared responses to time-pressure instructions, a manipulation with a mechanistic route to human judgment blocked for LLMs. Study 3 varied reasoning effort, as an analogue for time constrained human judgements. Studies 1B/2B alsovaried system prompt wording and numerical scale anchors. Results LLM judgments closely tracked human judgments for both stimulus sets, while responding in the same unanticipated purposivist direction in the new set as humans did. Sensitivity to text and purpose was robust across prompt variations. Responses to time-pressure instructions were model-specific, suggesting a distinction between conceptual competence and human alignment. Replication of the human pattern was most apparent in models with fewer parameters, and these effects were susceptible to prompt variation. Increasing reasoning effort produced no detectable change in rule application for most models though a significant purposivist trend was observed in higher-effort for GPT-oss and Claude Sonnet 5. Response variance remained lower for LLMs than humans despite our per-model temperature calibration to match human sample variance. Conclusions. Overall, the findings suggest that LLM rule application reflects a generalisable, standing semantic competence that does not typically depend on expanded deliberation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑