arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.04231cs.RO
收藏

GeoLanG: 基于几何的语言引导抓取与统一RGB-D多模态学习

GeoLanG: Geometry-Aware Language-Guided Grasping with Unified RGB-D Multimodal Learning

Rui Tang, Guankun Wang, Long Bai, Huxin Gao, Jiewen Lai, Chi Kit Ng, Jiazheng Wang, Fan Zhang, Hongliang Ren

AI总结:

GeoLanG通过统一RGB-D多模态学习,结合深度引导几何模块和自适应通道集成,实现鲁棒的语言引导抓取,提升复杂环境中的抓取精度和泛化能力。

摘要:

语言引导的抓取已成为一种有前景的范式,使机器人能够通过自然语言指令识别和操控目标物体,但在拥挤或遮挡的场景中仍然极具挑战性。现有方法通常依赖于多阶段流程,将物体感知和抓取分开,导致跨模态融合有限、计算冗余以及在拥挤、遮挡或低纹理场景中泛化能力差。为了解决这些限制,我们提出了GeoLanG,一种基于CLIP架构的端到端多任务框架,将视觉和语言输入统一到共享的表示空间中,以实现稳健的语义对齐和改进的泛化能力。为了在遮挡和低纹理条件下增强目标辨别能力,我们探索了更有效利用深度信息的方法,通过深度引导的几何模块(DGGM),将深度转换为显式的几何先验,并将其注入注意力机制中,而无需额外的计算开销。此外,我们提出了自适应密集通道集成,该方法自适应地平衡多层特征的贡献,以产生更具判别性和泛化能力的视觉表示。在OCID-VLG数据集以及仿真和真实硬件中的广泛实验表明,GeoLanG能够在复杂、拥挤的环境中实现精确且稳健的语言引导抓取,为更可靠的多模态机器人操作在真实的人类中心场景中铺平道路。

英文摘要:

Language-guided grasping has emerged as a promising paradigm for enabling robots to identify and manipulate target objects through natural language instructions, yet it remains highly challenging in cluttered or occluded scenes. Existing methods often rely on multi-stage pipelines that separate object perception and grasping, which leads to limited cross-modal fusion, redundant computation, and poor generalization in cluttered, occluded, or low-texture scenes. To address these limitations, we propose GeoLanG, an end-to-end multi-task framework built upon the CLIP architecture that unifies visual and linguistic inputs into a shared representation space for robust semantic alignment and improved generalization. To enhance target discrimination under occlusion and low-texture conditions, we explore a more effective use of depth information through the Depth-guided Geometric Module (DGGM), which converts depth into explicit geometric priors and injects them into the attention mechanism without additional computational overhead. In addition, we propose Adaptive Dense Channel Integration, which adaptively balances the contributions of multi-layer features to produce more discriminative and generalizable visual representations. Extensive experiments on the OCID-VLG dataset, as well as in both simulation and real-world hardware, demonstrate that GeoLanG enables precise and robust language-guided grasping in complex, cluttered environments, paving the way toward more reliable multimodal robotic manipulation in real-world human-centric settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑