本研究提出了一种基于大语言模型(LLM)和视觉语言模型(VLM)的智能多智能体框架,专门针对机器人领域进行优化。目标是将LLM和VLM的优势与计算工具相结合,自动分析和解决与机器人机械臂相关的问题。我们开发的框架能够接受文本和视觉输入,并能自动执行正向和反向运动学计算,计算关键点的速度和加速度,生成机器人的3D模拟,并最终在模拟环境中执行运动控制,所有这些操作均根据用户查询进行。为了评估该框架,设计了三个基准测试,每个测试包含十个问题。在第一个基准测试中,框架连接到GPT-4o、DeepSeek-V3.2和Claude-Sonnet-4.5及其对应的原始模型进行评估。目标是从文本描述中直接提取机器人的正向运动学。结果显示,与GPT-4o集成的框架在计算最终解决方案的准确性上最高,达到0.97,而单独的原始模型在相同任务上的准确性仅为0.30。同样,对于其他两个模型,该框架在准确性方面也始终优于对应的原始模型。第二个基准测试与第一个相同,只是输入以视觉形式提供。在此次测试中,GPT-4o LLM与Gemini 2.5 Pro VLM一起使用。结果显示,该框架在获得最终答案的准确性上达到0.93,比对应的原始模型高约20%。第三个基准测试涵盖了范围广泛的机器人任务,包括模拟、控制、速度和加速度计算,以及反向运动学和雅可比计算,该框架在这些任务上的准确性达到0.97。
英文摘要:
This study proposes an intelligent multi-agent framework built on LLMs and VLMs and specifically tailored to robotics. The goal is to integrate the strengths of LLMs and VLMs with computational tools to automatically analyze and solve problems related to robotic manipulators. Our developed framework accepts both textual and visual inputs and can automatically perform forward and inverse kinematics, compute velocities and accelerations of key points, generate 3D simulations of the robot, and ultimately execute motion control within the simulated environment, all according to the user's query. To evaluate the framework, three benchmark tests were designed, each consisting of ten questions. In the first benchmark test, the framework was evaluated while connected to GPT-4o, DeepSeek-V3.2, and Claude-Sonnet-4.5, as well as their corresponding raw models. The objective was to extract the forward kinematics of robots directly from textual descriptions. The results showed that the framework integrated with GPT-4o achieved the highest accuracy, reaching 0.97 in computing the final solution, whereas the raw model alone attained an accuracy of only 0.30 for the same task. Similarly, for the other two models, the framework consistently outperformed the corresponding raw models in terms of accuracy. The second benchmark test was identical to the first, except that the input was provided in visual form. In this test, the GPT-4o LLM was used alongside the Gemini 2.5 Pro VLM. The results showed that the framework achieved an accuracy of 0.93 in obtaining the final answer, which is approximately 20% higher than that of the corresponding raw model. The third benchmark test encompassed a range of robotic tasks, including simulation, control, velocity and acceleration computation, as well as inverse kinematics and Jacobian calculation, for which the framework achieved an accuracy of 0.97.
Design And Control of A Robotic Arm For Industrial Applications
CMR Technical Campus(CMR技术学院);JNTUH University College of Engineering, Science & Technology Hyderabad(贾瓦哈拉尔·尼赫鲁科技大学海得拉巴工程科学与技术学院);Holy Mary Institute of Technology(圣玛丽技术学院)
Fast Heuristic Scheduling and Trajectory Planning for Robotic Fruit Harvesters with Multiple Cartesian Arms
Department of Mechanical & Aerospace Engineering University of California, Davis(机械与航空航天工程系加州大学戴维斯分校);Department of Biological and Agricultural Engineering University of California, Davis(生物与农业工程系加州大学戴维斯分校)
Generating and Customizing Robotic Arm Trajectories using Neural Networks
Faculty of Mathematics, Physics and Informatics(数学、物理与信息学系);Comenius University Bratislava(布拉迪斯拉发康门尼大学);DIBRIS(迪布里斯);University of Genoa(热那亚大学);Italian Institute of Technology(意大利理工学院)
Multi-Objective Trajectory Planning for a Robotic Arm in Curtain Wall Installation
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Shenzhen University of Advanced Technology and University of Chinese Academy of Sciences(深圳先进技术大学和中国科学院大学)
D3-ARM: High-Dynamic, Dexterous and Fully Decoupled Cable-driven Robotic Arm
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院人工智能与机器人中心);School of Advanced Manufacturing, Sun Yat-sen University(中山大学先进制造学院)
Robotic-assisted Ultrasound for Fetal Imaging: Evolution from Single-arm to Dual-arm System
King’s College London(伦敦国王学院);Xtronics Ltd.(Xtronics有限公司);Guy’s & St Thomas’ NHS Foundation Trust(盖伊和圣托马斯NHS基金会信托);Evelina Children’s Hospital(伊芙琳娜儿童医院);Queen Mary University of London(伦敦玛丽女王大学)
Design and Control of a Novel Six-Degree-of-Freedom Hybrid Robotic Arm
Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences(北京农林科学院智能装备研究中心);School of Mechanical Electrical Engineering and Automation, Shanghai University(上海大学机电工程与自动化学院)
Continuous Collision Detection for a Robotic Arm Mounted on a Cable-Driven Parallel Robot
Tecnalia France(法国泰克纳利亚);LAAS-CNRS(法国国家科学研究中心自动化系统与机器人实验室);University of Toulouse(图卢兹大学);LIRMM(蒙彼利埃大学与法国国家科学研究中心联合机器人与微电子机械系统实验室);Université de Montpellier(蒙彼利埃大学);CNRS(法国国家科学研究中心)