Background. Evidence that large language models (LLMs) reproduce human judgments does not establish conceptual mastery: the correspondence may reflect memorisation or be sensitivite to incidental task features.
Objective. Across five experiments, we test whether 13 LLMs possess a generalisable competence in applying rules, including cases in which a rule's text and purpose point towards different outcomes.
Method. Study 1A compared LLM judgments with newly collected human data on published stimuli and matched vignettes created after the models' training cut-offs. Studies 2A/2B compared responses to time-pressure instructions, a manipulation with a mechanistic route to human judgment blocked for LLMs. Study 3 varied reasoning effort, as an analogue for time constrained human judgements. Studies 1B/2B alsovaried system prompt wording and numerical scale anchors.
Results LLM judgments closely tracked human judgments for both stimulus sets, while responding in the same unanticipated purposivist direction in the new set as humans did. Sensitivity to text and purpose was robust across prompt variations. Responses to time-pressure instructions were model-specific, suggesting a distinction between conceptual competence and human alignment. Replication of the human pattern was most apparent in models with fewer parameters, and these effects were susceptible to prompt variation. Increasing reasoning effort produced no detectable change in rule application for most models though a significant purposivist trend was observed in higher-effort for GPT-oss and Claude Sonnet 5. Response variance remained lower for LLMs than humans despite our per-model temperature calibration to match human sample variance.
Conclusions. Overall, the findings suggest that LLM rule application reflects a generalisable, standing semantic competence that does not typically depend on expanded deliberation.
SkillMaster: Toward Autonomous Skill Mastery in LLM Agents
Shandong University(山东大学);Zhongguancun Academy(中关村学院);Tsinghua University(清华大学);Southeast University(东南大学);University of Science and Technology of China(中国科学技术大学)
IntelliCode: A Multi-Agent LLM Tutoring System with Centralized Learner Modeling
School of Computer Science and Engineering, VIT-AP University(计算机科学与工程学院,VIT-AP大学);School of Electrical and Computer Sciences, Indian Institute of Technology Bhubaneswar(电气与计算机科学学院,印度理工学院布巴内斯瓦尔学院)
NL2GenSym: Natural Language to Generative Symbolic Rules for SOAR Cognitive Architecture via Large Language Models
College of Systems Engineering, National University of Defense Technology(系统工程学院,国防科技大学);State Key Laboratory of Digital Intelligent Modeling and Simulation(数字智能建模与仿真国家重点实验室);Test Center, National University of Defense Technology(测试中心,国防科技大学)
Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
Composing Verifiable Conceptual Models via Building Blocks: Towards Design-Time Verification of Agentic AI Workflows
Team EVERGREEN Inria Centre Inria d’Université Côte d’Azur(法国国家信息与自动化研究所蔚蓝海岸大学中心EVERGREEN团队);Department of Data Science William & Mary(威廉与玛丽学院数据科学系);Office of Enterprise Research and Innovation Old Dominion University(欧道明大学企业研究与创新办公室)
The Quantum Nature of Identity in Human Thought: Bose-Einstein Statistics for Conceptual Indistinguishability
Center Leo Apostel for Interdisciplinary Studies(利奥·阿波斯泰尔跨学科研究中心);Brussels Free University(布鲁塞尔自由大学);University of British Columbia(不列颠哥伦比亚大学);Instituto de Filosofía y Ciencias de la Complejidad (IFICC)(哲学与复杂性研究所(IFICC))