To enable humanoid robots to share our social space we need to develop technology for easy interaction with the robots using multiple modes such as speech, gestures and share our emotions with them. We have targeted this research towards addressing the core issue of emotion recognition problem which would require less computation resources and much lesser number of network hyperparameters which will be more adaptive to be computed on low resourced social robots for real time communication. More specifically, here we have proposed an Inception module based Convolutional Neural Network Architecture which has achieved improved accuracy of upto 6% improvement over the existing network architecture for emotion classification when combinedly tested over multiple datasets when tried over humanoid robots in real - time. Our proposed model is reducing the trainable Hyperparameters to an extent of 94% as compared to vanilla CNN model which clearly indicates that it can be used in real time based application such as human robot interaction. Rigorous experiments have been performed to validate our methodology which is sufficiently robust and could achieve high level of accuracy. Finally, the model is implemented in a humanoid robot, NAO in real time and robustness of the model is evaluated.
Touch and Tell: Multimodal Decoding of Human Emotions and Social Gestures for Robots
Faculty of Engineering and Architecture（工程与建筑学院）；IDLab-AIRO, Ghent University – imec（IDLab-AIRO，根特大学–imec）；Department of Engineering Science, University of Oxford（工程科学系，牛津大学）
Georgia Tech Center for Music Technology（佐治亚理工学院音乐技术中心）
Human collaboration with robotics is dependant on the development of a relationship between human and robot, without which performance and utilization can decrease. Emotion and personality conveyance has been shown to enhance robotic collaborations, with improved human-robot relationships and increased trust. One under-explored way for an artificial agent to convey emotions is through non-linguistic musical prosody. In this work we present a new 4.2 hour dataset of improvised emotional vocal phrases based on the Geneva Emotion Wheel. This dataset has been validated through extensive listening …
Technische Universität Kaiserslautern（凯泽斯劳滕工业大学）；Furhat Robotics（Furhat机器人公司）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）
Emotions are the intrinsic or extrinsic representations of our experiences. The importance of emotions during a human-human interaction is immense as it formulates the basis of our interaction framework. There are several approaches in psychology to evaluate emotional states in humans based on the perceived stimuli. However, the topic has been less explored as far as human-robot interaction is concerned. This paper uses an appropriate emotion appraisal mechanism from psychology, generating an emotional state in a humanoid robot on-the-fly during human-robot interaction. Since the exhibition of…
Social robots need intelligence in order to safely coexist and interact with humans. Robots without functional abilities in understanding others and unable to empathise might be a societal risk and they may lead to a society of socially impaired robots. In this work we provide a survey of three relevant human social disorders, namely autism, psychopathy and schizophrenia, as a means to gain a better understanding of social robots' future capability requirements. We provide evidence supporting the idea that social robots will require a combination of emotional intelligence and social intelligen…
In the next decade, social robots will be implemented in many public spaces to provide services to humans. We question the properties of these social robots to afford acceptance and spontaneous emotional interactions. More specifically, in the present study, we report the effects of idle motion frequency in a robot on emotional contagion in a face-to-face interactive task with a human participant. The robotic system Buddy was programmed to adopt a sad posture and facial expression while telling three sad stories and moving its head up/down at low, medium, and high frequency. Each participant (…
Human emotions are expressed through multiple modalities, including verbal and non-verbal information. Moreover, the affective states of human users can be the indicator for the level of engagement and successful interaction, suitable for the robot to use as a rewarding factor to optimize robotic behaviors through interaction. This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy and personalize emotional interaction for a human user. The goal is to apply this framework in social scenarios that can let…
Georgia Tech Center for Music Technology（佐治亚理工学院音乐技术中心）
As human-robot collaboration opportunities continue to expand, trust becomes ever more important for full engagement and utilization of robots. Affective trust, built on emotional relationship and interpersonal bonds is particularly critical as it is more resilient to mistakes and increases the willingness to collaborate. In this paper we present a novel model built on music-driven emotional prosody and gestures that encourages the perception of a robotic identity, designed to avoid uncanny valley. Symbolic musical phrases were generated and tagged with emotional information by human musicians…
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent's decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning effic…
Perception of Emotions in Human and Robot Faces: Is the Eye Region Enough?
Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；KTH Royal Institute of Technology（皇家理工学院（瑞典））；Furhat Robotics AB（Furhat机器人公司）；Donders Institute for Brain, Cognition and Behaviour（唐德斯脑与认知行为研究所）；Radboud University（拉德堡德大学）；Maastricht University（马斯特里赫特大学）
Georgia Tech Center for Music Technology（佐治亚理工学院音乐技术中心）
As robotic arms become prevalent in industry it is crucial to improve levels of trust from human collaborators. Low levels of trust in human-robot interaction can reduce overall performance and prevent full robot utilization. We investigated the potential benefits of using emotional musical prosody to allow the robot to respond emotionally to the user's actions. We tested participants' responses to interacting with a virtual robot arm that acted as a decision agent, helping participants select the next number in a sequence. We compared results from three versions of the application in a betwee…
Emotions are reactions that can be expressed through a variety of social signals. For example, anger can be expressed through a scowl, narrowed eyes, a long stare, or many other expressions. This complexity is problematic when attempting to recognize a human's expression in a human-robot interaction: categorical emotion models used in HRI typically use only a few prototypical classes, and do not cover the wide array of expressions in the wild. We propose a data-driven method towards increasing the number of known emotion classes present in human-robot interactions, to 28 classes or more. The m…
Emotional Expression in Low-Degrees-of-Freedom Robots: Assessing Perception with Reachy Mini
Robotic Musicianship Lab（机器人音乐性实验室）；Georgia Institute of Technology（佐治亚理工学院）；School of Computing and Communications（计算与通讯学院）；Ben-Gurion University of the Negev（贝内尔盖翁大学（内盖夫分校））；School of Brain Sciences and Cognition（脑科学与认知学院）；The Azrieli National Center for Autism and Neurodevelopment Research（阿兹里尔国家自闭症与神经发育研究中心）
Facial expressions are an ideal means of communicating one's emotions or intentions to others. This overview will focus on human facial expression recognition as well as robotic facial expression generation. In the case of human facial expression recognition, both facial expression recognition on predefined datasets as well as in real-time will be covered. For robotic facial expression generation, hand-coded and automated methods i.e., facial expressions of a robot are generated by moving the features (eyes, mouth) of the robot by hand-coding or automatically using machine learning techniques,…
Emotions can provide a natural communication modality to complement the existing multi-modal capabilities of social robots, such as text and speech, in many domains. We conducted three online studies with 112, 223, and 151 participants to investigate the benefits of using emotions as a communication modality for Search And Rescue (SAR) robots. In the first experiment, we investigated the feasibility of conveying information related to SAR situations through robots' emotions, resulting in mappings from SAR situations to emotions. The second study used Affect Control Theory as an alternative met…
The purpose of the present study is to learn emotion expression representations for artificial agents using reward shaping mechanisms. The approach takes inspiration from the TAMER framework for training a Multilayer Perceptron (MLP) to learn to express different emotions on the iCub robot in a human-robot interaction scenario. The robot uses a combination of a Convolutional Neural Network (CNN) and a Self-Organising Map (SOM) to recognise an emotion and then learns to express the same using the MLP. The objective is to teach a robot to respond adequately to the user's perception of emotions a…
Creating robots with emotional personalities will transform the usability of robots in the real world. As previous emotive social robots are mostly based on statically stable robots whose mobility is limited, this paper develops an animation to real world pipeline that enables dynamic bipedal robots that can twist, wiggle, and walk to behave with emotions. First, an animation method is introduced to design emotive motions for the virtual robot character. Second, a dynamics optimizer is used to convert the animated motion to dynamically feasible motion. Third, real time standing and walking con…
University of Plymouth（普利茅斯大学）；ANYbotics AG（ANYbotics股份公司）；Simon Fraser University（西蒙弗雷泽大学）；dSPACE GmbH（dSPACE有限公司）
In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration, since humans attribute, and perhaps subconsciously anticipate, such traces to perceive an agent as engaging, trustworthy, and socially present. Robotic emotional body language needs to be believable, nuanced and relevant to the context. We implemented a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions and can generate numerous new ones of similar believability and lifelikeness. The framew…
University of Denver（丹佛大学）；DreamFace Technologies, LLC（梦脸科技有限责任公司）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
This paper presents our recent research on integrating artificial emotional intelligence in a social robot (Ryan) and studies the robot's effectiveness in engaging older adults. Ryan is a socially assistive robot designed to provide companionship for older adults with depression and dementia through conversation. We used two versions of Ryan for our study, empathic and non-empathic. The empathic Ryan utilizes a multimodal emotion recognition algorithm and a multimodal emotion expression system. Using different input modalities for emotion, i.e. facial expression and speech sentiment, the empat…
Investigating the Effect of LED Signals and Emotional Displays in Human-Robot Shared Workspaces
German International University（德国国际大学）；Intelligent Autonomous Systems Lab（智能自主系统实验室）；Department of Computer Science, TU Darmstadt（计算机科学系，图恩-达姆施塔特技术大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, TU Darmstadt（认知科学中心，图恩-达姆施塔特技术大学）；Hessian Center for Artificial Intelligence (Hessian.AI)（黑森人工智能中心）
Towards Emotion Co-regulation with LLM-powered Socially Assistive Robots: Integrating LLM Prompts and Robotic Behaviors to Support Parent-Neurodivergent Child Dyads
Department of Industrial Design, Eindhoven University of Technology（工业设计系，埃因霍温理工大学）；Department of Biomedical Engineering, Eindhoven University of Technology（生物医学工程系，埃因霍温理工大学）；Department of Engineering, Institute of Science Tokyo（工程系，东京科学研究所）；Department of Engineering, Utrecht University（工程系，乌得勒支大学）
PERCY: Personal Emotional Robotic Conversational System
University of New South Wales（新南威尔士大学）；University of Technology Sydney（悉尼科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Universidad Central de Chile（智利中央大学）
A Hybrid Deep Learning Framework for Emotion Recognition in Children with Autism During NAO Robot-Mediated Interaction
Department of Information Technology, School of Engineering, Cochin University of Science and Technology（科钦大学科学与技术学院信息科技系）；Indian Institute of Technology Kanpur（印度理工学院坎普尔学院）
Socially assistive robots could help to support people's well-being in contexts such as art therapy where human therapists are scarce, by making art such as paintings together with people in a way that is emotionally contingent and creative. However, current art-making robots are typically either contingent, controlled as a tool by a human artist, or creative, programmed to paint independently, possibly because some complex and idiosyncratic concepts related to art, such as emotion and creativity, are not yet well understood. For example, the role of personalized taste in forming beauty evalua…
Emotive Response to a Hybrid-Face Robot and Translation to Consumer Social Robots
Imperial College London（帝国理工学院）；UK Dementia Research Institute（英国痴呆症研究所）；University of Bristol（布里斯托尔大学）；University of Warwick（华威大学）；Emotix Inc（Emotix公司）
Real-Time Imitation of Human Head Motions, Blinks and Emotions by Nao Robot: A Closed-Loop Approach
Human and Robot Interaction Lab, Electrical and Computer Engineering, University of Tehran（人类与机器人交互实验室，电气与计算机工程，法尔姆大学）；Department of Electrical Engineering, Shahrood University of Technology（电气工程系，沙霍德大学）；Intelligent Robotics Laboratory, Oakland University（智能机器人实验室，奥克兰大学）
University of Naples Federico II（那不勒斯费德里科二世大学）；Istituto Italiano di Tecnologia（意大利技术研究院）
The high request for autonomous and flexible HRI implies the necessity of deploying Machine Learning (ML) mechanisms in the robot control. Indeed, the use of ML techniques, such as Reinforcement Learning (RL), makes the robot behaviour, during the learning process, not transparent to the observing user. In this work, we proposed an emotional model to improve the transparency in RL tasks for human-robot collaborative scenarios. The architecture we propose supports the RL algorithm with an emotional model able to both receive human feedback and exhibit emotional responses based on the learning p…
University of South Carolina（南卡罗来纳大学）；University of Cincinnati（辛辛那提大学）；University of Wisconsin--Madison（威斯康星大学麦迪逊分校）；University of South Florida（佛罗里达州立大学）
Social robots like Moxie are designed to form strong emotional bonds with children, but their abrupt discontinuation can cause significant struggles and distress to children. When these services end, the resulting harm raises complex questions of who bears responsibility when children's emotional bonds are broken. Using the Moxie shutdown as a case study through a qualitative survey of 72 U.S. participants, our findings show that the responsibility is viewed as a shared duty across the robot company, parents, developers, and government. However, these attributions varied by political ideology …
Awakening Facial Emotional Expressions in Human-Robot
Institute of Machine Intelligence, University of Shanghai for Science and Technology（上海理工大学机器智能学院）；College Of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；Shanghai Droid Robot Co., Ltd.（上海机灵狗机器人有限公司）；Department of Informatics, University of Hamburg（汉堡大学信息学院）
In current youth-care programs, children with needs (mental health, family issues, learning disabilities, and autism) receive support from youth and family experts as one-to-one assistance at schools or hospitals. Occasionally, social robots have featured in such settings as support roles in a one-to-one interaction with the child. In this paper, we suggest the development of a symbiotic framework for real-time Emotional Support (ES) with social robots Knowledge Graphs (KG). By augmenting a domain-specific corpus from the literature on ES for children (between the age of 8 and 12) and providin…
Effectively recognising and applying emotions to interactions is a highly desirable trait for social robots. Implicitly understanding how subjects experience different kinds of actions and objects in the world is crucial for natural HRI interactions, with the possibility to perform positive actions and avoid negative actions. In this paper, we utilize the NICO robot's appearance and capabilities to give the NICO the ability to model a coherent affective association between a perceived auditory stimulus and a temporally asynchronous emotion expression. This is done by combining evaluations of e…
University of Hamburg（汉堡大学）；Department of Informatics, Knowledge Technology Institute（信息学系、知识技术研究所）
Speech emotion recognition (SER) is an important aspect of effective human-robot collaboration and received a lot of attention from the research community. For example, many neural network-based architectures were proposed recently and pushed the performance to a new level. However, the applicability of such neural SER models trained only on in-domain data to noisy conditions is currently under-researched. In this work, we evaluate the robustness of state-of-the-art neural acoustic emotion recognition models in human-robot interaction scenarios. We hypothesize that a robot's ego noise, room co…
Acoustically expressed emotions can make communication with a robot more efficient. Detecting emotions like anger could provide a clue for the robot indicating unsafe/undesired situations. Recently, several deep neural network-based models have been proposed which establish new state-of-the-art results in affective state evaluation. These models typically start processing at the end of each utterance, which not only requires a mechanism to detect the end of an utterance but also makes it difficult to use them in a real-time communication scenario, e.g. human-robot interaction. We propose the E…
