arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.18372cs.CV
收藏

你只需说一次即可看见

You Only Speak Once to See

Tianjin University(天津大学);University of Washington(华盛顿大学);University of Copenhagen(哥本哈根大学)

Wenhao Yang, Jianguo Wei, Wenhuan Lu, Lei Li

AI总结:

本文提出 YOSS 方法,通过对比学习与多模态对齐融合预训练音频和视觉模型,首次系统探索音频引导的物体定位,并验证其能提升定位精度与鲁棒性。

摘要:

利用视觉线索在图像中定位物体是计算机视觉中一种成熟的方法,然而音频作为一种用于物体识别与定位的模态,其潜力仍未得到充分探索。我们提出 YOSS,即“You Only Speak Once to See”,以利用音频在视觉场景中定位物体,称之为音频定位(Audio Grounding)。通过使用对比学习和多模态对齐将预训练的音频模型与视觉模型集成,我们的方法能够捕捉语音命令或描述,并将其直接映射到图像中对应的物体。实验结果表明,音频引导可以有效地应用于物体定位,说明引入音频引导可能提升当前物体定位方法的精度和鲁棒性,并改善机器人系统与计算机视觉应用的性能。这一发现为高级物体识别、场景理解以及开发更直观、能力更强的机器人系统开辟了新的可能性。

英文摘要:

Grounding objects in images using visual cues is a well-established approach in computer vision, yet the potential of audio as a modality for object recognition and grounding remains underexplored. We introduce YOSS, "You Only Speak Once to See," to leverage audio for grounding objects in visual scenes, termed Audio Grounding. By integrating pre-trained audio models with visual models using contrastive learning and multi-modal alignment, our approach captures speech commands or descriptions and maps them directly to corresponding objects within images. Experimental results indicate that audio guidance can be effectively applied to object grounding, suggesting that incorporating audio guidance may enhance the precision and robustness of current object grounding methods and improve the performance of robotic systems and computer vision applications. This finding opens new possibilities for advanced object recognition, scene understanding, and the development of more intuitive and capable robotic systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑