Large language models (LLMs) trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g.,from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions ("faster") depending on context (i.e., behavioral commonsense). This paper presents code as policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io
Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation
Australian Institute for Machine Learning(澳大利亚机器学习研究所);Adelaide University(阿德莱德大学);Responsible AI Research Centre(负责任人工智能研究中心);CSIRO Data61(联邦科学与工业研究组织数据61中心);The University of Queensland(昆士兰大学)
Speedup Patch: Learning a Plug-and-Play Policy to Accelerate Embodied Manipulation
National Key Laboratory for Novel Software Technology, Nanjing University, China;School of Artificial Intelligence, Nanjing University, China;Mila-Quebec AI Institute \& Université de Montréal, Canada
PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation
Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China(计算机科学与技术系,人工智能研究院,清华-博世联合机器学习中心,THBI实验室,BNRist中心,清华大学,北京,100084,中国);Peng Cheng Laboratory, 518108, China(鹏城实验室,518108,中国)
SpikingNav: Robust Embodied Navigation with Spiking Neural Policies
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);State Key Laboratory of Media Convergence and Communication, Communication University of China(中国传媒大学媒体融合与传播国家重点实验室);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Beijing National Research Center for Information Science and Technology, Tsinghua University(北京信息科学与技术国家研究中心)
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
School of Software, Harbin Institute of Technology(哈尔滨工业大学软件学院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus(赫瑞-瓦特大学马来西亚校区数学与计算机科学学院);School of Future Science and Engineering, Soochow University(苏州大学未来科学与工程学院)
CREATE, Aalborg University(奥尔堡大学CREATE);Department of Communication and Psychology, Aalborg University(奥尔堡大学传播与心理学系);CyPhy Life, Robotics and AI Lab, School of Science & Technology, IE University(IE大学科技学院CyPhy Life机器人与人工智能实验室)
LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos
Joy Future Academy, JD(京东探索研究院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Tsinghua SIGS, Tsinghua University(清华大学深圳国际研究生院);Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学院)
Inspect Robots: Evaluating the Capabilities and Safety of Embodied AI
Robocurve;University of Southern California(南加州大学);San Jose State University(圣何塞州立大学);Indian Institute of Technology Bhilai(印度理工学院比莱校区);University of Maryland(马里兰大学);Stony Brook University(石溪大学);Arizona State University(亚利桑那州立大学);Brown University(布朗大学)
X-DiffVLA: X-Embodied Diffusion Action Heads for Vision-Language-Action Models
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences(SKL-MAIS,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);BeingBeyond;School of Computer Science, Peking University(北京大学计算机学院)
Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies
University of South Florida(佛罗里达州立大学);Florida International University(佛罗里达国际大学);Johns Hopkins University(约翰霍普金斯大学);Nottingham Trent University(诺丁汉特伦特大学);Imperial College London(伦敦帝国理工学院)
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
S-Lab, Nanyang Technological University(新加坡南洋理工大学S-Lab);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);BUPT(北京邮电大学);XJTU(西安交通大学);Hong Kong Baptist University(香港浸会大学)
EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Cornell University(康奈尔大学);Universal Ubiquitous AI Co., Ltd.(Universal Ubiquitous AI有限公司);Hangzhou DEEP Robotics Technology Co., Ltd.(杭州深度机器人科技有限公司);National University of Singapore(新加坡国立大学)
Embodied AI: Emerging Risks and Opportunities for Policy Action
Centre for the Governance of AI(治理人工智能中心);French Center for AI Safety (CeSIA)(法国人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University of Bologna(博洛尼亚大学);Tony Blair Institute for Global Change(托尼·布莱尔全球变革研究所);Yale University(耶鲁大学);Yale Digital Ethics Center(耶鲁数字伦理中心)
Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models
Futian Laboratory(福田实验室);Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));School of Robotics, Hunan University(湖南大学机器人学院);South China University of Technology(华南理工大学);Visincept(Visincept公司);National Key Laboratory of Smart Farm Technologies and Systems(智能农业技术与系统国家重点实验室)
Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusion
Collaborative and Versatile Robots (CLOVER) Laboratory, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong(协作与多功能机器人实验室,T-Stone机器人研究所,香港中文大学,香港);Φ \Phi -Institute for Physical Human Intelligence(物理人机智能研究所);Center for Embodied Artificial Intelligence and Computer Vision, Shenzhen Loop Area Institute, Shenzhen, China(具身人工智能与计算机视觉中心,深圳环园研究院,深圳,中国);Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy(先进机器人系,意大利理工学院,热那亚,意大利)