Vision-language-model (VLM)-guided reinforcement learning (RL) has recently attracted significant attention for it, replacing brittle hand-crafted rewards with semantically grounded signals; however, deploying such simulation-trained policies on real vehicles remains a fundamental challenge, because they rely on simulator-native observations and simulator-coupled action semantics with no counterpart on physical hardware. We identify a general principle: the simulation-to-reality gap decomposes into two largely orthogonal axes, a sensing-and-dynamics domain gap and a task-and-geometry gap, the former closable without real-world policy training by re-projecting real perception and control onto the policy's training manifold. We formalize this as a transfer guarantee that bounds the deployment gap by three independently controllable error terms, and instantiate it as Sim2Real-AD, which combines a Geometric Observation Bridge, a Physics-Aware Action Mapping, a Two-Phase Progressive Training curriculum, and a Real-time Deployment Pipeline. As a proof of concept, a CARLA-trained VLM-guided RL policy is transferred zero-shot to a full-scale battery-electric Ford E-Transit van in Madison, WI, USA, and drives across car-following, obstacle-avoidance, and stop-sign scenarios using no real-world training data. To our knowledge, this is among the first zero-shot closed-loop deployments of a CARLA-trained VLM-guided RL policy on a full-scale real vehicle, and the decomposition offers a principled, broadly applicable route for moving simulation-trained, foundation-model-guided policies into the physical world, supporting energy-efficient intelligent driving on electrified transportation platforms. The demo video, code, and model checkpoint are available at: https://zilin-huang.github.io/Sim2Real-AD-website/.
Facebook AI Research（Facebook AI研究院）；University of California, Berkeley（加州大学伯克利分校）
Model-based reinforcement learning is a compelling framework for data-efficient learning of agents that interact with the world. This family of algorithms has many subcomponents that need to be carefully selected and tuned. As a result the entry-bar for researchers to approach the field and to deploy it in real-world tasks can be daunting. In this paper, we present MBRL-Lib -- a machine learning library for model-based reinforcement learning in continuous state-action spaces based on PyTorch. MBRL-Lib is designed as a platform for both researchers, to easily develop, debug and compare new algo…
Skalar Systems GmbH（斯卡拉系统有限责任公司）；FH JOANNEUM – University of Applied Sciences（约阿内高等专业学院）；Know-Center GmbH（诺中心有限责任公司）；Graz University of Technology（格拉茨技术大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
We investigate the feasibility of deploying Deep-Q based deep reinforcement learning agents to job-shop scheduling problems in the context of modular production facilities, using discrete event simulations for the environment. These environments are comprised of a source and sink for the parts to be processed, as well as (several) workstations. The agents are trained to schedule automated guided vehicles to transport the parts back and forth between those stations in an optimal fashion. Starting from a very simplistic setup, we increase the complexity of the environment and compare the agents'…
FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Models
School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；School of Software Engineering, South China University of Technology（软件工程学院，华南理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳校区）；Chinese University of Hong Kong（香港中文大学）
The aerial manipulator (AM) is a systematic operational robotic platform in high standard on algorithm robustness. Directly deploying the algorithms to the practical system will take numerous trial and error costs and even cause destructive results. In this paper, a new modular simulation platform is designed to evaluate aerial manipulation related algorithms before deploying. In addition, to realize a fully autonomous aerial grasping, a series of algorithm modules consisting a complete workflow are designed and integrated in the simulation platform, including perception, planning and control …
Visual-Language-Guided Task Planning for Horticultural Robots
the Siebel School of Computing and Data Science（塞比尔计算与数据科学学院）；the Department of Agricultural and Biological Engineering（农业与生物工程系）；National Center for Supercomputing Applications（国家超级计算中心）
CDE: Concept-Driven Exploration for Reinforcement Learning
Department of Electrical and Computer Engineering, Purdue University（电子工程系，普渡大学）；Department of Computer Science, Purdue University（计算机科学系，普渡大学）；Robotics Institute, Carnegie Mellon University（机器人研究所，卡内基梅隆大学）；Department of Management of Technology, EPFL（技术管理系，瑞士联邦理工学院）
Scalable, Training-Free Visual Language Robotics: A Modular Multi-Model Framework for Consumer-Grade GPUs
National Institute of Advanced Industrial Science and Technology (AIST)（日本先进工业科学技术研究院）；CNRS-AIST JRL (Joint Robotics Laboratory)（法国国家科学研究中心-日本先进工业科学技术研究院联合机器人实验室）
University of Pennsylvania（宾夕法尼亚大学）；Technical University of Munich（慕尼黑工业大学）；Lehigh University（理海大学）；University of Texas Austin（德克萨斯大学奥斯汀分校）；Poznan University of Technology（波兹南理工大学）；University of Virginia（弗吉尼亚大学）；Clemson University（克莱姆森大学）；Toyota Research Institute（丰田研究院）
Although robotics courses are well established in higher education, the courses often focus on theory and sometimes lack the systematic coverage of the techniques involved in developing, deploying, and applying software to real hardware. Additionally, most hardware platforms for robotics teaching are low-level toys aimed at younger students at middle-school levels. To address this gap, an autonomous vehicle hardware platform, called F1TENTH, is developed for teaching autonomous systems hands-on. This article describes the teaching modules and software stack for teaching at various educational …
