Embodied reference understanding is crucial for intelligent agents to predict referents based on human intention through gesture signals and language descriptions. This paper introduces the Attention-Dynamic DINO, a novel framework designed to mitigate misinterpretations of pointing gestures across various interaction contexts. Our approach integrates visual and textual features to simultaneously predict the target object's bounding box and the attention source in pointing gestures. Leveraging the distance-aware nature of nonverbal communication in visual perspective taking, we extend the virtual touch line mechanism and propose an attention-dynamic touch line to represent referring gesture based on interactive distances. The combination of this distance-aware approach and independent prediction of the attention source, enhances the alignment between objects and the gesture represented line. Extensive experiments on the YouRefIt dataset demonstrate the efficacy of our gesture information understanding method in significantly improving task performance. Our model achieves 76.4% accuracy at the 0.25 IoU threshold and, notably, surpasses human performance at the 0.75 IoU threshold, marking a first in this domain. Comparative experiments with distance-unaware understanding methods from previous research further validate the superiority of the Attention-Dynamic Touch Line across diverse contexts.
YouRefIt: Embodied Reference Understanding with Language and Gesture
University of California, Los Angeles(加州大学洛杉矶分校);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Peking University(北京大学);Tsinghua University(清华大学)
Technical Report of RoboSpatial Challenge at CVPR 2026: Selective Reasoning Activation and Reference-Frame Disambiguation for Embodied Spatial Reasoning
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Ruoyu Technology(若愚科技)
Understanding Embodied Reference with Touch-Line Transformer
Institute for AI Industry Research, Tsinghua University(清华大学智能产业研究院);UCSD(加州大学圣地亚哥分校);Intel Labs China(英特尔中国研究院);Peking University(北京大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限公司);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学)
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限责任公司)
Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding
Stony Brook University(石溪大学);BITS Pilani(pilani bits大学);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心);IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局信息技术研究院);APPCAIR;Massachusetts Institute of Technology(麻省理工学院);Singapore Management University(新加坡管理大学)
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Esslingen University of Applied Sciences(埃斯林根应用科学大学);Dr. Ing. h.c. F. Porsche AG(德意志联邦汽车工业联合会);University of Michigan(密歇根大学);Voxel51 Inc.(Voxel51公司)
MotionAdapter: Video Motion Transfer via Content-Aware Attention Customization
Hangzhou Dianzi University(杭州电子科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The Hong Kong University of Science and Technology(香港科技大学);Ocean University of China(中国海洋大学);Singapore Management University(新加坡管理大学)
Vision-language models lag human performance on physical dynamics and intent reasoning
East China Normal University(东华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);University of Agder(阿格德大学)