In this work, we highlight vulnerabilities in robotic systems integrating large language models (LLMs) and vision-language models (VLMs) due to input modality sensitivities. While LLM/VLM-controlled robots show impressive performance across various tasks, their reliability under slight input variations remains underexplored yet critical. These models are highly sensitive to instruction or perceptual input changes, which can trigger misalignment issues, leading to execution failures with severe real-world consequences. To study this issue, we analyze the misalignment-induced vulnerabilities within LLM/VLM-controlled robotic systems and present a mathematical formulation for failure modes arising from variations in input modalities. We propose empirical perturbation strategies to expose these vulnerabilities and validate their effectiveness through experiments on multiple robot manipulation tasks. Our results show that simple input perturbations reduce task execution success rates by 22.2% and 14.6% in two representative LLM/VLM-controlled robotic systems. These findings underscore the importance of input modality robustness and motivate further research to ensure the safe and reliable deployment of advanced LLM/VLM-controlled robotic systems.
T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models
School of Software Engineering, South China University of Technology（软件工程学院，华南理工大学）；School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学港大剑客基金会数据科学研究所）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电气与电子工程系）
Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models
Massachusetts Institute of Technology (MIT)（麻省理工学院）；MIT（麻省理工学院）；Google DeepMind（谷歌DeepMind）；Google, Paradigms of Intelligence（谷歌、范式智能）；Autodesk Research（Autodesk研究）；MIT Mechanical Engineering（麻省理工学院机械工程系）；MIT Architecture（麻省理工学院建筑系）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）；Yuandao AI（元岛人工智能）；Shanghai AI Lab（上海人工智能实验室）
FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Models
School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；School of Software Engineering, South China University of Technology（软件工程学院，华南理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
School of Intelligent Science and Technology, Nanjing University, China（南京大学智能科学与技术学院）；National Key Laboratory for Novel Software Technology, School of Artificial Intelligence, Nanjing University, China（南京大学新型软件技术国家实验室，人工智能学院）；School of Artificial Intelligence, Nanjing University, China（南京大学人工智能学院）；MACS Lab, University of Washington（华盛顿大学MACS实验室）
Reflectance Estimation for Proximity Sensing by Vision-Language Models: Utilizing Distributional Semantics for Low-Level Cognition in Robotics
College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学与工程学院）；College of Science and Engineering, Kanazawa University（金泽大学理工学院）
Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness
University of Maryland College Park（马里兰大学帕克分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Waterloo（滑铁卢大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Pennsylvania（宾夕法尼亚大学）；Amazon FAR（亚马逊 FAR）
Training-free Generation of Temporally Consistent Rewards from VLMs
Beijing Institute of Technology（北京理工大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Taobao & Tmall Group of Alibaba（阿里巴巴淘宝天猫集团）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
NVIDIA（英伟达）；University of Washington（华盛顿大学）；Universidad Católica San Pablo（圣巴勃罗天主教大学）；MIT（麻省理工学院）；Nanyang Technological University（南洋理工大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
ActivePose: Active 6D Object Pose Estimation and Tracking for Robotic Manipulation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工大学）；Shanghai Jiao Tong University（上海交通大学）；Istituto Italiano di Tecnologia（意大利理工学院）；The Hong Kong Polytechnic University（香港理工大学）；D-Robotics（D-机器人）
Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）
