Social visual behavior, as a type of non-verbal communication, plays a central role in studying social cognitive processes in interactive and complex settings of autism therapy interventions. However, for social visual behavior analytics in children with autism, it is challenging to collect gaze data manually and evaluate them because it costs a lot of time and effort for human coders. In this paper, we introduce a social visual behavior analytics approach by quantifying the mutual gaze performance of children receiving play-based autism interventions using an automated mutual gaze detection framework. Our analysis is based on a video dataset that captures and records social interactions between children with autism and their therapy trainers (N=28 observations, 84 video clips, 21 Hrs duration). The effectiveness of our framework was evaluated by comparing the mutual gaze ratio derived from the mutual gaze detection framework with the human-coded ratio values. We analyzed the mutual gaze frequency and duration across different therapy settings, activities, and sessions. We created mutual gaze-related measures for social visual behavior score prediction using multiple machine learning-based regression models. The results show that our method provides mutual gaze measures that reliably represent (or even replace) the human coders' hand-coded social gaze measures and effectively evaluates and predicts ASD children's social visual performance during the intervention. Our findings have implications for social interaction analysis in small-group behavior assessments in numerous co-located settings in (special) education and in the workplace.
Neurofeedback games are an effective and playful approach to enhance certain social and attentional capabilities in children with autism, which are promising to become widely accessible along with the commercialization of mobile EEG modules. However, little industry-based experiences are shared, regarding how to better design neurofeedback games to fine-tune their playability and user experiences for autistic children. In this paper, we review the experiences we gained from industry practice, in which a series of mobile EEG neurofeedback games have been developed for preschool autistic childre…
A good teaching method is incomprehensible for an autistic child. The autism spectrum disorder is a very diverse phenomenon. It is said that no two autistic children are the same. So, something that works for one child may not be fit for another. The same case is true for their education. Different children need to be approached with different teaching methods. But it is quite hard to identify the appropriate teaching method. As the term itself explains, the autism spectrum disorder is like a spectrum. There are multiple factors to determine the type of autism of a child. A child might even be…
Imitation plays an important role in development, being one of the precursors of social cognition. Even though some children with autism imitate spontaneously and other children with autism can learn to imitate, the dynamics of imitation is affected in the large majority of cases. Existing studies from the literature suggest that robots can be used to teach children with autism basic interaction skills like imitation. Based on these findings, in this study, we investigate if children with autism show more social engagement when interacting with an imitative robot (Fig 1) compared to a human pa…
Visual Programming Tools (VPTs) allow users to create interactive media projects such as games and animations using visual representations of programming concepts. Although VPTs have been shown to have huge potential for teaching children with cognitive impairments including those with Autism Spectrum Condition (ASC), research has shown that existing VPTs may not be accessible to them. Therefore, this study proposes a set of recommendations for the design of accessible VPTs for children with ASC. Recommendations were initially gathered and validated by interviewing experts (n=7). The interview…
In this exploratory study we assessed how attitudes of children with autism spectrum disorder (ASD) towards robots together with children's autism-related social impairments are linked to indicators of children's preference of an interaction with a robot over an interaction with a person. We found that children with ASD have overall positive attitudes towards robots and that they often prefer interacting with a robot than with a person. Several of children's attitudes were linked to children's longer gazes towards a robot compared to a person. Autism-related social impairments were linked to m…
Autism often remains undiagnosed in adolescents and adults. Prior research has indicated that an autistic individual often shows atypical fixation and gaze patterns. In this short paper, we demonstrate that by monitoring a user's gaze as they watch commonplace (i.e., not specialized, structured or coded) video, we can identify individuals with autism spectrum disorder. We recruited 35 autistic and 25 non-autistic individuals, and captured their gaze using an off-the-shelf eye tracker connected to a laptop. Within 15 seconds, our approach was 92.5% accurate at identifying individuals with an au…
Children with Autism Spectrum Disorder (ASD) experience deficits in verbal and nonverbal communication skills including motor control, turn-taking, and emotion recognition. Innovative technology, such as socially assistive robots, has shown to be a viable method for Autism therapy. This paper presents a novel robot-based music-therapy platform for modeling and improving the social responses and behaviors of children with ASD. Our autonomous social interactive system consists of three modules. We adopted Short-time Fourier Transform and Levenshtein distance to fulfill the design requirements: a…
Universidad Peruana Cayetano Heredia（秘鲁卡耶塔诺·埃雷迪亚大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Autism is a developmental disorder that affects social interaction and communication of children. The gold standard diagnostic tools are very difficult to use and time consuming. However, diagnostic could be deduced from child gaze preferences by looking a video with social and abstract scenes. In this work, we propose an algorithm based on convolutional neural networks to predict gaze direction for a fast and effective autism diagnosis. Early results show that our algorithm achieves real-time response and robust high accuracy for prediction of gaze direction.
There is a lack of autonomy on traditional Robot-Assisted Therapy systems interacting with children with autism. To overcome this limitation a supervised autonomous robot controller is being built. In this paper we present a multilayer reactive system within such controller. The goal of this Reactive system is to allow the robot to appropriately react to the child's behavior creating the illusion of being alive.
This research addressed whether interactions with social robots could facilitate joint attention of the autism spectrum disorder (ASD). Two conditions of initiators, namely 'Human' vs. 'Robot' were measured with 15 children with ASD and 15 age-matched typically developing (TD) children. Apart from fixation and gaze transition, a new longest common subsequence (LCS) approach was proposed to analyze eye-movement traces. Results revealed that children with ASD showed deficits of joint attention. Compared to the human agent, robot facilitate less fixations towards the targets, but it attracted mor…
Gaze-Informed Proactive AI Assistance for Children's Picture Exploration
Saarland University（萨尔兰大学）；Tübingen Digital Teaching Lab (TüDiLab)（蒂宾根数字教学实验室）；Leibniz-Institut für Wissensmedien (IWM Tübingen)（莱布尼茨知识媒体研究所）；Research Institute for Information Technology, Kyushu University（九州大学信息技术研究院）
University of Tehran（德黑兰大学）；Intelligent Systems Research Institute, SKKU（成均馆大学智能系统研究所）；Allameh Tabatabái University（阿拉梅塔巴塔巴伊大学）；Iran University of Medical Sciences（伊朗医科大学）；Shahid Beheshti University（沙希德贝赫什提大学）
Robot Assisted Therapy is a new paradigm in many therapies such as the therapy of children with autism spectrum disorder. In this paper we present the use of a parrot-like robot as an assistive tool in turn taking therapy. The therapy is designed in the form of a card game between a child with autism and a therapist or the robot. The intervention was implemented in a single subject study format and the effect sizes for different turn taking variables are calculated. The results show that the child robot interaction had larger effect size than the child trainer effect size in most of the turn t…
With most recent estimates giving an incidence rate of 1 in 68 children in the United States, the autism spectrum disorder (ASD) is a growing public health crisis. Many of these children struggle to make eye contact, recognize facial expressions, and engage in social interactions. Today the standard for treatment of the core autism-related deficits focuses on a form of behavior training known as Applied Behavioral Analysis. To address perceived deficits in expression recognition, ABA approaches routinely involve the use of prompts such as flash cards for repetitive emotion recognition training…
Affect-aware socially assistive robotics (SAR) tutors have great potential to augment and democratize professional therapeutic interventions for children with autism spectrum disorders (ASD) from different socioeconomic backgrounds. However, the majority of research on SAR for ASD has been on teaching cognitive and/or social skills, not on addressing users' emotional needs for real-world social situations. To bridge that gap, this work aims to develop personalized affect-aware SAR tutors to help alleviate social anxiety and foster social grit-the growth mindset for social skill development-in …
Autism spectrum disorder (ASD) is a developmental disorder that influences the communication and social behavior of a person in a way that those in the spectrum have difficulty in perceiving other people's facial expressions, as well as presenting and communicating emotions and affect via their own faces and bodies. Some efforts have been made to predict and improve children with ASD's affect states in play therapy, a common method to improve children's social skills via play and games. However, many previous works only used pre-trained models on benchmark emotion datasets and failed to consid…
Introducing CALMED: Multimodal Annotated Dataset for Emotion Detection in Children with Autism
University of Galway（戈尔韦大学）；LORIA CNRS/INRIA/Université de Lorraine（洛林大学国家科学研究中心/法国国家信息与自动化研究所洛林计算机科学及其应用实验室）；Université Sorbonne Paris Nord（巴黎索邦北大学）；ASK - All Special Kids（ASK-所有特殊儿童）
National University of Singapore（新加坡国立大学）；Ministry of Education, Singapore（新加坡教育部）
This paper reviews the use of socially interactive robots to assist in the therapy of children with autism. The extent to which the robots were successful in helping the children in their social, emotional, and communication deficits was investigated. Child-robot interactions were scrutinized with respect to the different target behaviors that are to be elicited from a child during therapy. These behaviors were thoroughly examined with respect to a childs development needs. Most importantly, experimental data from the surveyed works were extracted and analyzed in terms of the target behaviors …
Technische Universität München（慕尼黑工业大学）；University of Cambridge（剑桥大学）
Individuals with Autism Spectrum Conditions (ASC) have marked difficulties using verbal and non-verbal communication for social interaction. The running ASC-Inclusion project aims to help children with ASC by allowing them to learn how emotions can be expressed and recognised via playing games in a virtual world. The platform includes analysis of users' gestures, facial, and vocal expressions using standard microphone and web-cam or a depth sensor, training through games, text communication with peers, animation, video and audio clips. We present the state of play in realising such a serious g…
In recent years, much focus has been put on employing technology to make novel behavioural aids for those with autism. Most of these are digital adaptations of tools used in standard behavioural therapy to enforce normative skills. These digital counterparts are often used outside of both the larger therapeutic context and the real world, in which the learned skills might apply. To address this, we are designing a system of automatic expression recognition on wearable devices that integrates directly into the families daily social interactions, to give children and their caregivers the tools a…
University of Southern California（南加州大学）；Massachusetts Institute of Technology（麻省理工学院）
Affect-aware socially assistive robotics (SAR) has shown great potential for augmenting interventions for children with autism spectrum disorders (ASD). However, current SAR cannot yet perceive the unique and diverse set of atypical cognitive-affective behaviors from children with ASD in an automatic and personalized fashion in long-term (multi-session) real-world interactions. To bridge this gap, this work designed and validated personalized models of arousal and valence for children with ASD using a multi-session in-home dataset of SAR interventions. By training machine learning (ML) algorit…
Socially assistive robotics (SAR) has great potential to provide accessible, affordable, and personalized therapeutic interventions for children with autism spectrum disorders (ASD). However, human-robot interaction (HRI) methods are still limited in their ability to autonomously recognize and respond to behavioral cues, especially in atypical users and everyday settings. This work applies supervised machine learning algorithms to model user engagement in the context of long-term, in-home SAR interventions for children with ASD. Specifically, we present two types of engagement models for each …
Robots have great potential to facilitate future therapies for children on the autism spectrum. However, existing robots lack the ability to automatically perceive and respond to human affect, which is necessary for establishing and maintaining engaging interactions. Moreover, their inference challenge is made harder by the fact that many individuals with autism have atypical and unusually diverse styles of expressing their affective-cognitive states. To tackle the heterogeneity in behavioral cues of children with autism, we use the latest advances in deep learning to formulate a personalized …
The Nexus of AR/VR, AI, UI/UX, and Robotics Technologies in Enhancing Learning and Social Interaction for Children with Autism Spectrum Disorders: A Systematic Review
Exploring Gaze Pattern Differences Between Autistic and Neurotypical Children: Clustering, Visualisation, and Prediction
Singapore University of Technology and Design（新加坡科技设计大学）；A*STAR Institute for Infocomm Research（新加坡资讯通信研究院）；University of Wisconsin Madison（威斯康星大学麦迪逊分校）
Autism Spectrum Disorder (ASD) is the area where many researches enduring like Magnetic Resonance Imaging (MRI), called diffusion tensor imaging, Early Start Denver Model (ESDM) to provide an easier life for the people diagnosed. After years and years of combined funding sources from public and private funding, these researches show great promises in recent years. In this paper, we have tried to show a way how children with Down Syndrome Autism can learn through game therapy. These game therapies have shown an immense number of improvements among those children to learn alphabets along with de…
Exploring Image Transforms derived from Eye Gaze Variables for Progressive Autism Diagnosis
College of Engineering and Information Technology, University of Dubai, UAE（工程与信息技术学院，迪拜大学）；School of Electrical, Computer, and Telecommunications Engineering, University of Wollongong, Australia（电气、计算机与电信工程学院，沃尔longong大学）；College of Medicine and Health Sciences, Mohammed Bin Rashid University Dubai, UAE（医学与健康科学学院，迪拜穆罕默德·本·拉希德大学）
Development of Immersive Virtual and Augmented Reality-Based Joint Attention Training Platform for Children with Autism
Indian Institute of Technology Delhi（印度德里理工学院）；Indian Institute of Technology Jodhpur（印度焦浦尔理工学院）；All India Institute of Medical Sciences Delhi（印度医学科学院德里分院）
University of Hawaiʻi at Mānoa（夏威夷大学马诺阿分校）；Stanford University School of Medicine（斯坦福大学医学院）
Autism Spectrum Disorder (autism) is a neurodevelopmental delay which affects at least 1 in 44 children. Like many neurological disorder phenotypes, the diagnostic features are observable, can be tracked over time, and can be managed or even eliminated through proper therapy and treatments. Yet, there are major bottlenecks in the diagnostic, therapeutic, and longitudinal tracking pipelines for autism and related delays, creating an opportunity for novel data science solutions to augment and transform existing workflows and provide access to services for more affected families. Several prior ef…
A multimodal large language model for evidence-based autism spectrum disorder screening
Zhejiang Normal University（浙江师范大学）；State Grid Corporation of China（国家电网有限公司）；Huzhou Normal University（湖州师范学院）；Zhejiang University of Technology（浙江工业大学）
Empirical Study of Gaze Behavior in Children and Young Adults Using Deep Neural Networks and Robot Implementation: A Comparative Analysis of Social Situations
Sharif University of Technology（谢里夫理工大学）；Islamic Azad University（伊斯兰阿扎德大学）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；University of Reading（雷丁大学）；De Montfort University（德蒙福特大学）；Sangath（桑加斯）
Screening for any of the Autism Spectrum Disorders is a complicated process often involving a hybrid of behavioural observations and questionnaire based tests. Typically carried out in a controlled setting, this process requires trained clinicians or psychiatrists for such assessments. Riding on the wave of technical advancement in mobile platforms, several attempts have been made at incorporating such assessments on mobile and tablet devices. In this paper we analyse videos generated using one such screening test. This paper reports the first use of the efficacy of using the observer's distan…
Eye contact is a crucial element of non-verbal communication that signifies interest, attention, and participation in social interactions. As a result, measures of eye contact arise in a variety of applications such as the assessment of the social communication skills of children at risk for developmental disorders such as autism, or the analysis of turn-taking and social roles during group meetings. However, the automated measurement of visual attention during naturalistic social interactions is challenging due to the difficulty of estimating a subject's looking direction from video. This pap…
The coronavirus disease (COVID-19) pandemic affected our lives deeply, just like everyone else, the children also suffered from the restrictions due to COVID-19 affecting their education and social interactions with others, being restricted from play areas and schools for a long time. Although social robots provide a promising solution to support children in their education, healthcare and social interaction with others, the precautions due to COVID-19 also introduced new constraints in the social robotics research. In this paper, we will discuss the benefits and challenges encountered in chil…
EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR Activities
University of Waterloo（滑铁卢大学）；Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National Research Council（国家研究委员会）
We have developed a system for automatic facial expression recognition, which runs on Google Glass and delivers real-time social cues to the wearer. We evaluate the system as a behavioral aid for children with Autism Spectrum Disorder (ASD), who can greatly benefit from real-time non-invasive emotional cues and are more sensitive to sensory input than neurotypically developing children. In addition, we present a mobile application that enables users of the wearable aid to review their videos along with auto-curated emotional information on the video playback bar. This integrates our learning a…
Beyond Usability: A UX Case Study on Using "Withdrawal Design" to Challenge Engagement Metrics in Social Robotics
Tsinghua University（清华大学）；Nanjing University of Chinese Medicine（南京中医药大学）；Nanjing University（南京大学）；University of Washington（华盛顿大学）；North China University of Science and Technology Health Science Center（华北理工大学健康科学中心）
Syracuse University（雪城大学）；UnityPoint Health（UnityPoint健康）；Stevens Institute of Technology（史蒂文斯理工学院）；Binghamton University（宾汉姆顿大学）；Texas Tech University（得克萨斯理工大学）
Application of humanoid robots has been common in the field of healthcare and education. It has been recurrently used to improve social behavior and mollify distress level among children with autism, cancer and cerebral palsy. This article discusses the same from a human factors perspective. It shows how people of different age and gender have a different opinion towards the application and acceptance of humanoid robots. Additionally, this article highlights the influence of cerebral condition and social interaction on a user behavior and attitude towards humanoid robots. Our study performed a…
Surprising Performances of Students with Autism in Classroom with NAO Robot
School of Computer and Information Technology（计算机与信息学院）；Northeast Petroleum University（东北石油大学）；School of Computer Science and Engineering（计算机科学与工程学院）；Changshu Institute of Technology（常州职业技术学院）；Changshu Special Education School（常州特殊教育学校）；School of Chinese Language and Literature（中文语言文学学院）；Nanjing Normal University（南京师范大学）
Deep Learning Based Approach to Enhanced Recognition of Emotions and Behavioral Patterns of Autistic Children
Department of Information Technology（信息科技系）；Sri Lanka Institute of Information Technology（斯里兰卡信息科技研究所）；Department of Computer Systems Engineering（计算机系统工程系）
In current youth-care programs, children with needs (mental health, family issues, learning disabilities, and autism) receive support from youth and family experts as one-to-one assistance at schools or hospitals. Occasionally, social robots have featured in such settings as support roles in a one-to-one interaction with the child. In this paper, we suggest the development of a symbiotic framework for real-time Emotional Support (ES) with social robots Knowledge Graphs (KG). By augmenting a domain-specific corpus from the literature on ES for children (between the age of 8 and 12) and providin…
