Understanding Embodied Reference with Touch-Line Transformer
Institute for AI Industry Research, Tsinghua University(清华大学智能产业研究院);UCSD(加州大学圣地亚哥分校);Intel Labs China(英特尔中国研究院);Peking University(北京大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
We study embodied reference understanding, the task of locating referents using embodied gestural signals and language references. Human studies have revealed that objects referred to or pointed to do not lie on the elbow-wrist line, a common misconception; instead, they lie on the so-called virtual touch line. However, existing human pose representations fail to incorporate the virtual touch line. To tackle this problem, we devise the touch-line transformer: It takes as input tokenized visual and textual features and simultaneously predicts the referent's bounding box and a touch-line vector. Leveraging this touch-line prior, we further devise a geometric consistency loss that encourages the co-linearity between referents and touch lines. Using the touch-line as gestural information improves model performances significantly. Experiments on the YouRefIt dataset show our method achieves a +25.0% accuracy improvement under the 0.75 IoU criterion, closing 63.6% of the gap between model and human performances. Furthermore, we computationally verify prior human studies by showing that computational models more accurately locate referents when using the virtual touch line than when using the elbow-wrist line.
YouRefIt: Embodied Reference Understanding with Language and Gesture
University of California, Los Angeles(加州大学洛杉矶分校);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Peking University(北京大学);Tsinghua University(清华大学)
Technical Report of RoboSpatial Challenge at CVPR 2026: Selective Reasoning Activation and Reference-Frame Disambiguation for Embodied Spatial Reasoning
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Ruoyu Technology(若愚科技)
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限公司);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学)
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限责任公司)
Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding
Stony Brook University(石溪大学);BITS Pilani(pilani bits大学);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心);IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局信息技术研究院);APPCAIR;Massachusetts Institute of Technology(麻省理工学院);Singapore Management University(新加坡管理大学)
Embodied Image Quality Assessment for Robotic Intelligence
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(图像通信与网络工程研究所,上海交通大学);College of Information Science and Technology, Beijing University of Chemical Technology(信息科学与技术学院,北京化工大学);USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University(USC-SJTU文化产业与创意产业研究所,上海交通大学)
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Esslingen University of Applied Sciences(埃斯林根应用科学大学);Dr. Ing. h.c. F. Porsche AG(德意志联邦汽车工业联合会);University of Michigan(密歇根大学);Voxel51 Inc.(Voxel51公司)
Vision-language models lag human performance on physical dynamics and intent reasoning
East China Normal University(东华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);University of Agder(阿格德大学)