Grounding Before Generalizing: How AI Differs from Humans in Causal Transfer
Department of Automation(清华大学自动化系);Institute for Artificial Intelligence(北京大学人工智能研究院);School of Psychological and Cognitive Sciences(北京大学心理与认知科学系);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Beijing Key Laboratory of Behavior and Mental Health(北京行为与心理健康重点实验室)
Extracting abstract causal structures and applying them to novel situations is a hallmark of human intelligence. While Large Language Models (LLMs) and Vision Language Models (VLMs) have shown strong performance on a wide range of reasoning tasks, their capacity for interactive causal learning -- inducing latent structures through sequential exploration and transferring them across contexts -- remains uncharacterized. Human learners accomplish such transfer after minimal exposure, whereas classical Reinforcement Learning (RL) agents fail catastrophically. Whether state-of-the-art Artificial Intelligence (AI) models possess human-like mechanisms for abstract causal structure transfer is an open question. Using the OpenLock paradigm requiring sequential discovery of Common Cause (CC) and Common Effect (CE) structures, here we show that models exhibit fundamentally delayed or absent transfer: even successful models require initial environmental-specific mapping -- what we term environmental grounding -- before efficiency gains emerge, whereas humans leverage prior structural knowledge from the very first solution attempt. In the text-only condition, models matched or exceeded human discovery efficiency. In contrast, visual information -- in both the image-only and text-and-image conditions -- overall degraded rather than enhanced performance, revealing a broad reliance on symbolic processing rather than integrated multimodal reasoning. Models further exhibited systematic CC/CE asymmetries absent in humans, suggesting heuristic biases rather than direction-neutral causal abstraction. These findings reveal that large-scale statistical learning does not produce the decontextualized causal schemas underpinning human analogical reasoning, establishing grounding-dependent transfer as a fundamental limitation of current LLMs and VLMs.
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);East China University of Science and Technology(东华大学)
The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans
Department of Informatics(信息学院);Athens University of Economics and Business(雅典经济与商业大学);Archimedes, Athena Research Center(阿基米德·雅典研究中心);Instituto de Telecomunicações(电信研究所);Department of Computer Science(计算机科学系);Columbia University(哥伦比亚大学)
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);Intelligent Game and Decision Lab, Academy of Military Sciences(智能游戏与决策实验室,军事科学院)
HawkEye: Training Video-Text LLMs for Grounding Text in Videos
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
Efficient Adaptation For Remote Sensing Visual Grounding
American University of Beirut(贝鲁特美国大学);Lebanese University(黎巴嫩大学);RASID SARL;WASDI;Institute for Applied Mathematics, CNR Bari(应用数学研究所,CNR巴里);National Center for Remote Sensing, CNRS(遥感国家中心,CNRS)
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Massachusetts Institute of Technology(麻省理工学院);Shanghai Jiao Tong University(上海交通大学);MIT(麻省理工学院);UMass Amherst(马萨诸塞大学阿默斯特分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding
University of Macau(澳门大学);Shandong University(山东大学);Northwestern Polytechnical University(西北工业大学);Tsinghua University(清华大学);Shanghai AI Laboratory(上海人工智能实验室)
Grounding Text Embeddings in Stakeholder Associations
University of Oxford(牛津大学);Institute for Wicked Problems(复杂问题研究所);The Chinese University of Hong Kong(香港中文大学);Danish Technical University(丹麦技术大学);Aarhus University(奥胡斯大学)