CognitiveDog：基于大型多模态模型将视觉与语言转化为四足机器人动作的系统
CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Quadruped Robot
AI总结：
本文提出CognitiveDog系统，将大型多模态模型部署于配备夹爪的Unitree Go1四足机器人上，结合Visual-SLAM实现基于自然语言指令的自主物体操作与任务执行。
摘要：
本文介绍了CognitiveDog，这是一项四足机器人的开创性开发成果，搭载大型多模态模型（LMM），不仅能够与人类进行语言交流，还能通过物体操作与环境进行物理交互。该系统在配备定制夹爪的Unitree Go1机器狗上实现，并展示了自主决策能力，能够独立确定最合适的动作以及与各种物体的交互方式，以完成用户定义的任务。这些任务不一定包含直接指令，要求机器人基于自然语言输入和环境线索来理解并执行任务。本文深入探讨了该系统的复杂性、数据集特征以及软件架构。这一开发的关键在于机器人利用Visual-SLAM进行空间导航的能力、有效操纵和搬运物体的能力，以及在任务执行过程中提供有洞察力的自然语言解说能力。实验结果凸显了该机器人先进的任务理解能力和适应性，强调了其在现实世界应用中的潜力。用于微调机器狗行为生成模型的数据集可在以下链接获取：huggingface.co/datasets/ArtemLykov/CognitiveDog_dataset。
英文摘要：
This paper introduces CognitiveDog, a pioneering development of quadruped robot with Large Multi-modal Model (LMM) that is capable of not only communicating with humans verbally but also physically interacting with the environment through object manipulation. The system was realized on Unitree Go1 robot-dog equipped with a custom gripper and demonstrated autonomous decision-making capabilities, independently determining the most appropriate actions and interactions with various objects to fulfill user-defined tasks. These tasks do not necessarily include direct instructions, challenging the robot to comprehend and execute them based on natural language input and environmental cues. The paper delves into the intricacies of this system, dataset characteristics, and the software architecture. Key to this development is the robot's proficiency in navigating space using Visual-SLAM, effectively manipulating and transporting objects, and providing insightful natural language commentary during task execution. Experimental results highlight the robot's advanced task comprehension and adaptability, underscoring its potential in real-world applications. The dataset used to fine-tune the robot-dog behavior generation model is provided at the following link: huggingface.co/datasets/ArtemLykov/CognitiveDog_dataset
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
