arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20219cs.ROcs.AI
收藏

一种结合视频和语音的大型语言模型在人机交互中的应用方法

An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction

Dalian University of Technology(大连理工大学)

Guanting Shen, Zi Tian

AI总结:

本文提出结合视觉-语言模型、语音处理和模糊逻辑的多模态人机交互框架,通过语音指令实现机械臂的精准操控,实验显示命令执行准确率达75%,为未来人机协作提供灵活的基础。

摘要:

在人机交互(HRI)中,准确解读人类意图是一个核心挑战,也是实现人机之间更自然和直观协作的关键要求。本文提出了一种新的多模态HRI框架,结合先进的视觉-语言模型、语音处理和模糊逻辑,以实现对Dobot Magician机械臂的精确和自适应控制。所提出的系统集成了Florence-2进行物体检测,Llama 3.1进行自然语言理解,以及Whisper进行语音识别,为用户提供了一种无缝且直观的接口,通过语音命令进行物体操控。通过同时解决场景感知和动作规划,该方法增强了命令解读和执行的可靠性。在消费级硬件上进行的实验评估显示,命令执行的准确率为75%,突显了系统稳健性和适应性。除了当前性能外,所提出的架构为未来HRI研究提供了一个灵活且可扩展的基础,通过紧密耦合的语音和视觉-语言处理,提供了一条通往更复杂和自然人机协作的实用途径。

英文摘要:

Interpreting human intent accurately is a central challenge in human-robot interaction (HRI) and a key requirement for achieving more natural and intuitive collaboration between humans and machines. This work presents a novel multimodal HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic to enable precise and adaptive control of a Dobot Magician robotic arm. The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition, providing users with a seamless and intuitive interface for object manipulation through spoken commands. By jointly addressing scene perception and action planning, the approach enhances the reliability of command interpretation and execution. Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75\%, highlighting both the robustness and adaptability of the system. Beyond its current performance, the proposed architecture serves as a flexible and extensible foundation for future HRI research, offering a practical pathway toward more sophisticated and natural human-robot collaboration through tightly coupled speech and vision-language processing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑