Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an "Analyzer" that interprets natural language to discern key task-related concepts, (ii) a "Planner" that generates comprehensive strategies based on the language input and key concepts, (iii) a "Calculator" that computes parameters for each skill, and (iv) a "Coder" that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend explicit or implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems. Demos are available on our project page: https://creative-robotool.github.io/.
A Framework for Collaborating a Large Language Model Tool in Brainstorming for Triggering Creative Thoughts
R.B. Annis School of Engineering(R.B.安尼斯工程学院);University of Indianapolis(印第安纳波利斯大学);School of Journalism and Strategic Communication(新闻与战略传播学院);Ball State University(鲍尔州立大学)
Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification
University of Science and Technology of China(中国科学技术大学);Institute of Computing Technology, CAS(中国科学院计算技术研究所);Cambricon Technologies(寒武纪科技);Intelligent Software Research Center, Institute of Software, CAS(中国科学院软件研究所智能软件研究中心);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Innovation Center for Processor Technologies(上海处理器技术创新中心)
ORIBA: Exploring LLM-Driven Role-Play Chatbot as a Creativity Support Tool for Original Character Artists
Computer Science Research Centre, Royal College of Art(皇家艺术学院计算机科学研究中心);Digital Media, School of Literature, Media, and Communication, Georgia Institute of Technology(佐治亚理工学院数字媒体系);Princeton University(普林斯顿大学);Digital Media, Georgia Institute of Technology(佐治亚理工学院数字媒体系);Division of Integrative Systems and Design, The Hong Kong University of Science and Technology(香港科学大学整合系统与设计 division);Industrial Design Department, College of Engineering, KAIST(韩国科学技术院工程学院工业设计系)
Towards Path-Creative Navigation: Robot Navigation through Embodied Interaction
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);College of Information Science and Technology, Eastern Institute of Technology(东方理工大学信息科学与技术学院);School of Mechanical and Power Engineering, Zhengzhou university(郑州大学机械与动力工程学院);University of Waterloo(滑铁卢大学)
Alter-Art: Exploring Embodied Artistic Creation through a Robot Avatar
Soft Robotics for Human Cooperation and Rehabilitation(人类协作与康复软机器人);Istituto Italiano di Tecnologia(意大利技术研究院);Dipartimento di Ingegneria dell’Informazione(信息工程学院)
Pluri-perspectivism in Human-robot Co-creativity with Older Adults
Amsterdam University of Applied Sciences, The Netherlands(荷兰应用科学大学);Leiden University, The Netherlands(莱顿大学);University of Amsterdam, The Netherlands(阿姆斯特丹大学)
CREATE, Aalborg University(奥尔堡大学CREATE);Department of Communication and Psychology, Aalborg University(奥尔堡大学传播与心理学系);CyPhy Life, Robotics and AI Lab, School of Science & Technology, IE University(IE大学科技学院CyPhy Life机器人与人工智能实验室)
Shaping Expressiveness in Robotics: The Role of Design Tools in Crafting Embodied Robot Movements
CHArt Laboratory, Department of Psychology, University Paris8(巴黎第八大学心理学系CHArt实验室);Lab of INIT Robots, Department of Mechanical Engineering, École de technologie supérieure(École de technologie supérieure机械工程系INIT机器人实验室);Department of Dance, Université du Québec à Montréal(魁北克大学蒙特利尔分校舞蹈系);Costech Laboratory, Department of Humanities and Social Sciences, Université technologique de Compiègne(Compiègne技术大学人文与社会科学系Costech实验室)