Large Language Models (LLMs) trained using massive text datasets have recently shown promise in generating action plans for robotic agents from high level text queries. However, these models typically do not consider the robot's environment, resulting in generated plans that may not actually be executable, due to ambiguities in the planned actions or environmental constraints. In this paper, we propose an approach to generate environmentally-aware action plans that agents are better able to execute. Our approach involves integrating environmental objects and object relations as additional inputs into LLM action plan generation to provide the system with an awareness of its surroundings, resulting in plans where each generated action is mapped to objects present in the scene. We also design a novel scoring function that, along with generating the action steps and associating them with objects, helps the system disambiguate among object instances and take into account their states. We evaluated our approach using the VirtualHome simulator and the ActivityPrograms knowledge base and found that action plans generated from our system had a 310% improvement in executability and a 147% improvement in correctness over prior work. The complete code and a demo of our method is publicly available at https://github.com/hri-ironlab/scene_aware_language_planner.
TITAN: Graph-Executable Reasoning for Cyber Threat Intelligence
Sapienza Università di Roma(萨皮恩扎罗马大学);Institute of Informatics and Telematics (IIT)(信息与电信学研究院);National Research Council of Italy (CNR)(意大利国家研究委员会);Department of Excellence in Robotics and AI(机器人与人工智能卓越部门);TeCIP Institute(TeCIP研究所);Scuola Superiore Sant'Anna(圣安娜高级学校)
AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation
School of Software Engineering, South China University of Technology(软件工程学院,华南理工大学);School of Future Technology, South China University of Technology(未来技术学院,华南理工大学);Shien-Ming Wu School of Intelligent Engineering, South China University of Technology(智能工程学院,华南理工大学)
From Evidence to Trajectory: Abductive Reasoning Path Synthesis for Retrieval-Augmented Generation Agents Development
The Chinese University of Hong Kong, Sha Tin, NT, Hong Kong(香港中文大学);Université de Montréal, Montréal, Quebéc, Canada(蒙特利尔大学);McGill University, Montréal, Quebéc, Canada(麦吉尔大学);Mila - Quebéc AI Institute, Montréal, Quebéc, Canada(魁北克AI研究院);Huawei Noah’s Ark Lab, Montréal, Quebéc, Canada(华为诺亚实验室)
Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Industry-education Integration, University of Chinese Academy of Sciences(中国科学院大学产教融合学院)
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
S-Lab, Nanyang Technological University(新加坡南洋理工大学S-Lab);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);BUPT(北京邮电大学);XJTU(西安交通大学);Hong Kong Baptist University(香港浸会大学)
AgentSUMO: An Agentic Framework for Interactive Simulation Scenario Generation in SUMO via Large Language Models
Graduate School of Data Science, Korea Advanced Institute of Science and Technology(数据科学研究生院,韩国科学技术院);Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology(土木与环境工程系,韩国科学技术院)
Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning
Technische Universität Darmstadt(达姆施塔特工业大学);Center for Mind, Brain and Behavior, Uni. Marburg and JLU Giessen(马尔堡大学与吉森尤斯图斯·李比希大学心智、大脑与行为研究中心);Amazon Alexa(亚马逊Alexa)
Benchmark for Planning and Control with Large Language Model Agents: Blocksworld with Model Context Protocol
Institute of Automation Technology, Helmut Schmidt University / University of the Federal Armed Forces Hamburg, Germany(自动化技术研究所,海姆·施密特大学/联邦武装部队大学汉堡,德国);Siemens AG, Nuremberg, Germany(西门子股份公司,纽伦堡,德国)
TC-IDM: Grounding Video Generation for Executable Zero-shot Robot Motion
Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);State Key Laboratory of Multimedia Information Processing(多媒体信息处理国家重点实验室);School of Computer Science, Peking University(北京大学计算机科学学院)