The significant advancements in embodied vision navigation have raised concerns about its susceptibility to adversarial attacks exploiting deep neural networks. Investigating the adversarial robustness of embodied vision navigation is crucial, especially given the threat of 3D physical attacks that could pose risks to human safety. However, existing attack methods for embodied vision navigation often lack physical feasibility due to challenges in transferring digital perturbations into the physical world. Moreover, current physical attacks for object detection struggle to achieve both multi-view effectiveness and visual naturalness in navigation scenarios. To address this, we propose a practical attack method for embodied navigation by attaching adversarial patches to objects, where both opacity and textures are learnable. Specifically, to ensure effectiveness across varying viewpoints, we employ a multi-view optimization strategy based on object-aware sampling, which optimizes the patch's texture based on feedback from the vision-based perception model used in navigation. To make the patch inconspicuous to human observers, we introduce a two-stage opacity optimization mechanism, in which opacity is fine-tuned after texture optimization. Experimental results demonstrate that our adversarial patches decrease the navigation success rate by an average of 22.39%, outperforming previous methods in practicality, effectiveness, and naturalness. Code is available at: https://github.com/chen37058/Physical-Attacks-in-Embodied-Nav
Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);UAS Esslingen(埃森嫩大学);TU Wien(维也纳技术大学);Dr. Ing. h.c. F. Porsche AG(保时捷股份有限公司);University of Michigan(密歇根大学);Voxel51 Inc.(Voxel51公司)
Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection
LAAS-CNRS, Universite de Toulouse, CNRS(法国国家科学研究中心图卢兹分部LAAS、图卢兹大学、法国国家科学研究中心);Department of Electrical Engineering and Automation at Aalto University(阿尔托大学电气工程与自动化系)
VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
Adelaide University(阿德莱德大学);Responsible AI Research Centre, Australian Institute for Machine Learning(负责任人工智能研究中心、澳大利亚机器学习研究所);Tongji University(同济大学);ManyCore;Zhejiang University(浙江大学);Shanghai AI Lab(上海人工智能实验室);CSIRO Data61
IndustryNav: Evaluating Spatial Reasoning and Safety of Embodied Agents in Dynamic Industrial Navigation
Michigan State University(密歇根州立大学);Independent Researcher(独立研究者);University of California, Irvine(加州大学尔湾分校);Ohio State University(俄亥俄州立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Santa Barbara(加州大学圣巴巴拉分校);University of Pittsburgh(匹兹堡大学);University of Virginia(弗吉尼亚大学);Arizona State University(亚利桑那州立大学);Purdue University Northwest(普渡大学西北分校)
Beijing Institute of Technology(北京理工大学);ETH Zurich(苏黎世联邦理工学院);University of Technology Sydney(悉尼科技大学);Yangtze Delta Region Academy of Beijing Institute of Technology(北京理工大学长三角研究院)
NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)
RoboTidy : A 3D Gaussian Splatting Household Tidying Benchmark for Embodied Navigation and Action
Huazhong University of Science and Technology(华中科技大学);The University of Hong Kong(香港大学);INFIFORCE Intelligent Technology Co., Ltd.(INFIFORCE智能技术有限公司);Zhejiang University(浙江大学);Guangming Lab, Shenzhen(深圳光明实验室)
FloNa: Floor Plan Guided Embodied Visual Navigation
Beijing Institute of Technology(北京理工大学);Yangtze Delta Region Academy of Beijing Institute of Technology(北京理工大学长三角研究院);Beijing Racobit Electronic Information Technology Co., Ltd.(北京睿科比特电子信息技术有限公司)
MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing(福建智能感知与计算重点实验室);Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(多媒体可信感知与高效计算重点实验室);Beihang University(北航);Nanyang Technological University(南洋理工大学);University of Chinese Academy of Sciences(中国科学院大学);Zhongguancun Academy(中关村学院)
Meta AI Research(Meta人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
像图像-目标导航这样的现实长期任务涉及探索和利用阶段。由于目标的图像,一个具体的代理必须探索发现目标,即,使用学习的先验知识进行有效搜索。一旦目标被发现,智能体必须准确地校准到目标的最后一英里的导航。与任何强大的系统一样,探索性目标发现和利用性最后一英里导航之间的切换可以更好地从错误中恢复。根据这些直观的导轨,我们提出了SLING,以提高现有的图像目标导航系统的性能。作为对现有方法的补充,我们专注于最后一英里导航,并利用神经描述符来利用问题的底层几何结构。通过简单但有效的开关,我们可以轻松地将SLING与启发式、强化学习和神经模块化策略联系起来。在标准化的图像目标导航基准(Hahn et al. 2021)上,我们提高了策略,场景和情节复杂性的性能,将最先进的成功率从45%提高到55%。除了真实感模拟,我们在三个物理场景中进行了真实的机器人实验,并发现这些改进可以很好地转移到真实环境中。
Australian National University(澳大利亚国立大学);Princeton University(普林斯顿大学);Carnegie Mellon University(卡内基梅隆大学);Intel Labs(英特尔实验室);UC Berkeley(加州大学伯克利分校);George Mason University(乔治梅森大学);Facebook(脸书);Allen Institute for AI(艾伦人工智能研究所)
Consistent Attack: Universal Adversarial Perturbation on Embodied Vision Navigation
Department of Computer Science and Technology, Beijing National Research Center for Information Science and Technology, Tsinghua-Bosch Joint Center for Machine Learning, Institute for Artificial Intelligence, Tsinghua University, Beijing 100084, China(计算机科学与技术系,北京信息科学国家研究中心,清华-博世联合机器学习中心,人工智能研究院,清华大学,北京)
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside(加州大学河滨分校可信自主系统实验室);University of Michigan(密歇根大学);Workday;University of Southern California(南加州大学);Texas A&M University(德克萨斯A&M大学);Lehigh University(莱斯大学)
NavBench: Probing Multimodal Large Language Models for Embodied Navigation
The University of Adelaide(阿德莱德大学);The University of Queensland(昆士兰大学);CSIRO Data61(澳大利亚联邦科学与工业研究组织数据61研究中心);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Tongji University(同济大学)