Interpreting human intent accurately is a central challenge in human-robot interaction (HRI) and a key requirement for achieving more natural and intuitive collaboration between humans and machines. This work presents a novel multimodal HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic to enable precise and adaptive control of a Dobot Magician robotic arm. The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition, providing users with a seamless and intuitive interface for object manipulation through spoken commands. By jointly addressing scene perception and action planning, the approach enhances the reliability of command interpretation and execution. Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75\%, highlighting both the robustness and adaptability of the system. Beyond its current performance, the proposed architecture serves as a flexible and extensible foundation for future HRI research, offering a practical pathway toward more sophisticated and natural human-robot collaboration through tightly coupled speech and vision-language processing.
A Social Robot with Inner Speech for Dietary Guidance
TheEngineRoom, Department of Informatics Bioengineering, Robotics and System Engineering, University of Genoa, Genoa, Italy(TheEngineRoom,信息生物工程与机器人系统工程系,热那亚大学,热那亚,意大利);Department of Engineering, University of Palermo, Palermo, Italy(工程系,巴勒莫大学,巴勒莫,意大利)
Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody
IT University of Copenhagen(丹麦技术大学);University of Florida(佛罗里达大学);Brown University(布朗大学);Intelligent Agents Research Group(智能代理研究组);Intelligent Robot Lab(智能机器人实验室)
EmojiVoice: Towards long-term controllable expressivity in robot speech
School of Computing Science, Simon Fraser University(计算科学学院,西蒙弗雷泽大学);SUPMICROTECH, CNRS, Institut FEMTO-ST, Université de Franche-Comté(SUPMICROTECH、法国国家科学研究中心、FEMTO-ST研究所、弗朗什-孔泰大学);Enchanted Tools;Division of Speech Music and Hearing, KTH Royal Institute of Technology(语音音乐与听觉系,皇家理工学院)
Evaluating Speech-in-Speech Perception via a Humanoid Robot
University Medical Center Groningen(格罗宁根大学医学中心);University of Groningen(格罗宁根大学);Pento Audiology Centre(Pento听力中心);Lyon Neuroscience Research Center(里昂神经科学研究中心);Université Claude Bernard Lyon 1(里昂第一大学);Université de Lyon(里昂大学)
Towards engagement models that consider individual factors in HRI: on the relation of extroversion and negative attitude towards robots to gaze and speech during a human-robot assembly task
Inria(法国国家信息与自动化研究所);Loria(洛林计算机科学及其应用实验室);CNRS(法国国家科学研究中心);Université de Lorraine(洛林大学);TU Darmstadt(达姆施塔特工业大学);LIP6(LIP6实验室);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Sorbonne Universités(索邦大学);UPMC Université Paris 06(巴黎第六大学);Institut des Systèmes Intelligents et de Robotique (ISIR)(智能系统与机器人研究所);Université Paris 8(巴黎第八大学)
WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
University of Engineering and Technology, Vietnam National University(越南国立大学工程与技术大学);School of Information Science, Japan Advanced Institute of Science and Technology(日本先端科学技术大学院大学信息科学学院);School of Electronics and Computer Science, University of Southampton(南安普顿大学电子与计算机科学学院);Department of Robotics, Hanyang University(汉阳大学机器人学系)
Design and implementation of audio communication system for social-humanoid robot Lumen as an exhibition guide in Electrical Engineering Days 2015
Bandung Institute of Technology(万隆理工学院)
社会机器人流明是一个人形机器人创建的行为像人类和人类的朋友。在这项研究中,Lumen场景仅限于Lumen作为2015年电气工程日的展览指南,万隆理工学院电气工程本科生和研究生的研讨会和展览。作为一个展览指南,Lumen配备了Nao机器人,服务器和处理应用程序。音频通信系统就是其中的一种处理应用。该系统的目的是创建语言交流,使Lumen能够接收人类的声音并自然地对其做出反应。为了能够像人类一样进行交流,音频通信系统由语音识别模块将语音数据转换为文本,语音合成器模块将文本数据转换为语音,以及性别识别模块区分成年女性和男性声音。语音识别模块采用Google语音识别API实现,语音合成模块采用Acapela引擎实现,性别识别模块采用快速傅立叶变换算法提取的语音信号特征实现。用于实现的硬件是Nao机器人、计算机和无线调制解调器。- Lumen Robot Sosial Robot是一种能够像人类一样工作并为人类服务的类人机器人。音频通讯系统是一个应用最广泛的照明系统,它能够控制人体的亮度,并对自然产生响应,比如人体对自然的响应。为了实现人类等通讯,通讯音频系统需要一个强大的模块:语音识别来提供数据支持,语音合成器来提供数据支持,性别识别来提供想要和想要的东西。
Factored Reasoning with Inner Speech and Persistent Memory for Evidence-Grounded Human-Robot Interaction
Department of Engineering, University of Palermo(帕尔米罗大学工程系);TheEngineRoom, Department of Informatics Bioengineering, Robotics and System Engineering(TheEngineRoom,信息生物工程、机器人与系统工程系)
ERR@HRI 2024 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Interactions
Politecnico di Milano(米兰理工大学);University of Cambridge(剑桥大学);Cornell University(康奈尔大学);Johns Hopkins University(约翰斯·霍普金斯大学);Cankaya University(恰卡亚大学);Cornell Tech(康奈尔科技学院)
Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop
DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室);University of Maryland(马里兰大学);University of Texas at El Paso(德克萨斯大学埃尔帕索分校);University of California at Los Angeles(加利福尼亚大学洛杉矶分校);Cornell University(康奈尔大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Michigan(密歇根大学);Microsoft(微软公司);University of Rochester(罗切斯特大学);Boise State University(博伊西州立大学);DFKI(德国人工智能研究中心);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Semio(Semio公司);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Sheffield(谢菲尔德大学);University of Washington(华盛顿大学);Mount Holyoke College(曼荷莲学院);Carnegie Mellon University(卡内基梅隆大学);Tufts University(塔夫茨大学);Adobe(奥多比公司);Brown University(布朗大学);USC Institute for Creative Technologies(南加州大学创意技术研究所);University of California at Davis(加利福尼亚大学戴维斯分校)
What Can You Say to a Robot? Capability Communication Leads to More Natural Conversations
Vrije Universiteit Amsterdam(瓦赫宁根大学阿姆斯特丹);Utrecht University(乌得勒支大学);Delft University of Technology(代尔夫特理工大学);KTH Royal Institute of Technology(皇家理工学院)