The deep supervised and reinforcement learning paradigms (among others) have the potential to endow interactive multimodal social robots with the ability of acquiring skills autonomously. But it is still not very clear yet how they can be best deployed in real world applications. As a step in this direction, we propose a deep learning-based approach for efficiently training a humanoid robot to play multimodal games---and use the game of `Noughts & Crosses' with two variants as a case study. Its minimum requirements for learning to perceive and interact are based on a few hundred example images, a few example multimodal dialogues and physical demonstrations of robot manipulation, and automatic simulations. In addition, we propose novel algorithms for robust visual game tracking and for competitive policy learning with high winning rates, which substantially outperform DQN-based baselines. While an automatic evaluation shows evidence that the proposed approach can be easily extended to new games with competitive robot behaviours, a human evaluation with 130 humans playing with the Pepper robot confirms that highly accurate visual perception is required for successful game play.
Human emotions are expressed through multiple modalities, including verbal and non-verbal information. Moreover, the affective states of human users can be the indicator for the level of engagement and successful interaction, suitable for the robot to use as a rewarding factor to optimize robotic behaviors through interaction. This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy and personalize emotional interaction for a human user. The goal is to apply this framework in social scenarios that can let…
GSON: A Group-based Social Navigation Framework with Large Multimodal Model
Center for Artificial Intelligence and Robotics, Tsinghua Shenzhen International Graduate School（人工智能与机器人中心，清华大学深圳国际研究生院）；School of Computing, National University of Singapore（computing 学院，新加坡国立大学）
A Multimodal Neural Network for Recognizing Subjective Self-Disclosure Towards Social Robots
Amazon（亚马逊）；School of Psychology and Neuroscience, University of Glasgow（心理学与神经科学学院，格拉斯哥大学）；Ben-Gurion University of the Negev（内盖夫本·古里安大学）；University of Cambridge（剑桥大学）；ETH Zurich（苏黎世联邦理工学院）
Touch and Tell: Multimodal Decoding of Human Emotions and Social Gestures for Robots
Faculty of Engineering and Architecture（工程与建筑学院）；IDLab-AIRO, Ghent University – imec（IDLab-AIRO，根特大学–imec）；Department of Engineering Science, University of Oxford（工程科学系，牛津大学）
Mobile telepresence robots (MTRs) have become increasingly popular in the expanding world of remote work, providing new avenues for people to actively participate in activities at a distance. However, humans operating MTRs often have difficulty navigating in densely populated environments due to limited situation awareness and narrow field-of-view, which reduces user acceptance and satisfaction. Shared autonomy in navigation has been studied primarily in static environments or in situations where only one pedestrian interacts with the robot. We present a multimodal shared autonomy approach, le…
RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants
Singapore University of Technology and Design（新加坡科技设计大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Nanyang Technological University（南洋理工大学）；Chongqing University（重庆大学）
It is crucial for any assistive robot to prioritize the autonomy of the user. For a robot working in a task setting to effectively maintain a user's autonomy it must provide timely assistance and make accurate decisions. We use four independent high-precision, low-recall models, a mutual gaze model, task model, confirmatory gaze model, and a lexical model, that predict a user's need for assistance. Improving upon our four independent models, we used a sliding window method and a random forest classification algorithm to capture temporal dependencies and fuse the independent models with a late …
For robots to coexist with humans in a social world like ours, it is crucial that they possess human-like social interaction skills. Programming a robot to possess such skills is a challenging task. In this paper, we propose a Multimodal Deep Q-Network (MDQN) to enable a robot to learn human-like interaction skills through a trial and error method. This paper aims to develop a robot that gathers data during its interaction with a human and learns human interaction behaviour from the high-dimensional sensory information using end-to-end reinforcement learning. This paper demonstrates that the r…
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expressions and gestures from multimodal videos. Recent multimodal methods have set the state of the art on many tasks, but have difficulty modeling the complex face-to-face conversational dynamics across speaking turns in social interaction, particularly in a self-supervised setup. In this paper, we propose Face-to-Face Contrastive Learning (F2F-CL), a graph neural network designed to model social interactions using fact…
CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Games
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
Social media is daily creating massive multimedia content with paired image and text, presenting the pressing need to automate the vision and language understanding for various multimodal classification tasks. Compared to the commonly researched visual-lingual data, social media posts tend to exhibit more implicit image-text relations. To better glue the cross-modal semantics therein, we capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity. Afterwards, the classification tasks are explored via self-training in a teacher-student f…
Dataminr Inc.（Dataminr公司）；Rutgers University（罗格斯大学）；University of California, Davis（加州大学戴维斯分校）
Recent developments in image classification and natural language processing, coupled with the rapid growth in social media usage, have enabled fundamental advances in detecting breaking events around the world in real-time. Emergency response is one such area that stands to gain from these advances. By processing billions of texts and images a minute, events can be automatically detected to enable emergency response workers to better assess rapidly evolving situations and deploy resources accordingly. To date, most event detection techniques in this area have focused on image-only or text-only…
Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations
Seoul National University（首尔国立大学）；Human-Computer Interaction Institute, Carnegie Mellon University（人机交互研究所，卡内基梅隆大学）；Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学）
University of Maryland, College Park（马里兰大学帕克分校）；ShareChat
We present 3MASSIV, a multilingual, multimodal and multi-aspect, expertly-annotated dataset of diverse short videos extracted from short-video social media platform - Moj. 3MASSIV comprises of 50k short videos (20 seconds average duration) and 100K unlabeled videos in 11 different languages and captures popular short video trends like pranks, fails, romance, comedy expressed via unique audio-visual formats like self-shot videos, reaction videos, lip-synching, self-sung songs, etc. 3MASSIV presents an opportunity for multimodal and multilingual semantic understanding on these unique videos by a…
Southern University of Science and Technology（南方科技大学）；University of Birmingham（伯明翰大学）
Pedestrian trajectory forecasting is a fundamental task in multiple utility areas, such as self-driving, autonomous robots, and surveillance systems. The future trajectory forecasting is multi-modal, influenced by physical interaction with scene contexts and intricate social interactions among pedestrians. The mainly existing literature learns representations of social interactions by deep learning networks, while the explicit interaction patterns are not utilized. Different interaction patterns, such as following or collision avoiding, will generate different trends of next movement, thus, th…
Zhejiang University（浙江大学）；University of Southern California（南加州大学）
Multimodal pre-training breaks down the modality barriers and allows the individual modalities to be mutually augmented with information, resulting in significant advances in representation learning. However, graph modality, as a very general and important form of data, cannot be easily interacted with other modalities because of its non-regular nature. In this paper, we propose MMGA (Multimodal learning with Graph Alignment), a novel multimodal pre-training framework to incorporate information from graph (social network), image and text modalities on social media to enhance user representatio…
TIB – Leibniz Information Centre for Science and Technology（莱布尼茨科技信息中心（TIB））；L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）；Jozef Stefan Institute（约瑟夫·斯特凡研究所）
In recent years, the problem of misinformation on the web has become widespread across languages, countries, and various social media platforms. Although there has been much work on automated fake news detection, the role of images and their variety are not well explored. In this paper, we investigate the roles of image and text at an earlier stage of the fake news detection pipeline, called claim detection. For this purpose, we introduce a novel dataset, MM-Claims, which consists of tweets and corresponding images over three topics: COVID-19, Climate Change and broadly Technology. The dataset…
MuSeD: A Multimodal Spanish Dataset for Sexism Detection in Social Media Videos
University of Barcelona, CLiC-Language and Computing Center（巴塞罗那大学，CLiC语言与计算中心）；University of Copenhagen, Department of Computer Science（哥本哈根大学，计算机科学系）
VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（武汉大学计算机学院多媒体软件国家工程研究中心）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
People post their opinions and experiences on social media, yielding rich databases of end-users' sentiments. This paper shows to what extent machine learning can analyze and structure these databases. An automated data analysis pipeline is deployed to provide insights into user-generated content for researchers in other domains. First, the domain expert can select an image and a term of interest. Then, the pipeline uses image retrieval to find all images showing similar content and applies aspect-based sentiment analysis to outline users' opinions about the selected term. As part of an interd…
Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline
School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）；School of Computer Science and Technology, Northwestern Polytechnical University（计算机科学与技术学院，西北工业大学）
Recognizing emotions during social interactions has many potential applications with the popularization of low-cost mobile sensors, but a challenge remains with the lack of naturalistic affective interaction data. Most existing emotion datasets do not support studying idiosyncratic emotions arising in the wild as they were collected in constrained environments. Therefore, studying emotions in the context of social interactions requires a novel dataset, and K-EmoCon is such a multimodal dataset with comprehensive annotations of continuous emotions during naturalistic conversations. The dataset …
