Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding
Stony Brook University(石溪大学);BITS Pilani(pilani bits大学);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心);IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局信息技术研究院);APPCAIR;Massachusetts Institute of Technology(麻省理工学院);Singapore Management University(新加坡管理大学)
Atharv Mahesh Mane, Dulanga Weerakoon, Vigneshwaran Subbaraju, Sougata Sen, Sanjay E. Sarma, Archan Misra
3-Dimensional Embodied Reference Understanding (3D-ERU) combines a language description and an accompanying pointing gesture to identify the most relevant target object in a 3D scene. Although prior work has explored pure language-based 3D grounding, there has been limited exploration of 3D-ERU, which also incorporates human pointing gestures. To address this gap, we introduce a data augmentation framework-Imputer, and use it to curate a new benchmark dataset-ImputeRefer for 3D-ERU, by incorporating human pointing gestures into existing 3D scene datasets that only contain language instructions. We also propose Ges3ViG, a novel model for 3D-ERU that achieves ~30% improvement in accuracy as compared to other 3D-ERU models and ~9% compared to other purely language-based 3D grounding models. Our code and dataset are available at https://github.com/AtharvMane/Ges3ViG.
YouRefIt: Embodied Reference Understanding with Language and Gesture
University of California, Los Angeles(加州大学洛杉矶分校);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Peking University(北京大学);Tsinghua University(清华大学)
Can Vision Language Models Understand Mimed Actions?
Information Sciences Institute(信息科学研究所);Institute for Creative Technologies(创意技术研究所);Department of Computer Science(计算机科学系);University of Southern California(南加州大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Aristotle University of Thessaloniki(希腊雅典纳大学)
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限责任公司)
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室OpenGVLab);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Nanjing University(南京大学);SenseTime Research(商汤科技研究院);The University of Hong Kong(香港大学);Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学)
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);KIT Campus Transfer GmbH (KCT)(KIT校园转移有限公司);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学)
VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection
Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Dalian University of Technology(大连理工大学)