A Computational Model of Early Word Learning from the Infant's Point of View
Luddy School of Informatics, Computing, and Engineering, Indiana University（印第安纳大学卢迪信息、计算与工程学院）；Department of Psychological and Brain Sciences, Indiana University（印第安纳大学心理与脑科学系）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Human infants have the remarkable ability to learn the associations between object names and visual objects from inherently ambiguous experiences. Researchers in cognitive science and developmental psychology have built formal models that implement in-principle learning algorithms, and then used pre-selected and pre-cleaned datasets to test the abilities of the models to find statistical regularities in the input data. In contrast to previous modeling approaches, the present study used egocentric video and gaze data collected from infant learners during natural toy play with their parents. This allowed us to capture the learning environment from the perspective of the learner's own point of view. We then used a Convolutional Neural Network (CNN) model to process sensory data from the infant's point of view and learn name-object associations from scratch. As the first model that takes raw egocentric video to simulate infant word learning, the present study provides a proof of principle that the problem of early word learning can be solved, using actual visual data perceived by infant learners. Moreover, we conducted simulation experiments to systematically determine how visual, perceptual, and attentional properties of infants' sensory experiences may affect word learning.
University of Cambridge（剑桥大学）；University of Glasgow（格拉斯哥大学）
Infants' neurological development is heavily influenced by their motor skills. Evaluating a baby's movements is key to understanding possible risks of developmental disorders in their growth. Previous research in psychology has shown that measuring specific movements or gestures such as face touches in babies is essential to analyse how babies understand themselves and their context. This research proposes the first automatic approach that detects face touches from video recordings by tracking infants' movements and gestures. The study uses a multimodal feature fusion approach mixing spatial a…
Roux Institute, Northeastern University（东北大学鲁克斯研究所）；Northeastern University（东北大学）
We apply computer vision pose estimation techniques developed expressly for the data-scarce infant domain to the study of torticollis, a common condition in infants for which early identification and treatment is critical. Specifically, we use a combination of facial landmark and body joint estimation techniques designed for infants to estimate a range of geometric measures pertaining to face and upper body symmetry, drawn from an array of sources in the physical therapy and ophthalmology research literature in torticollis. We gauge performance with a range of metrics and show that the estimat…
Infant motion analysis is a topic with critical importance in early childhood development studies. However, while the applications of human pose estimation have become more and more broad, models trained on large-scale adult pose datasets are barely successful in estimating infant poses due to the significant differences in their body ratio and the versatility of their poses. Moreover, the privacy and security considerations hinder the availability of adequate infant pose data required for training of a robust model from scratch. To address this problem, this paper presents (1) building and pu…
Deep learning-based neurodevelopmental assessment in preterm infants
Image Computing Laboratory（图像计算实验室）；Shaanxi University of Science and Technology（陕西科技大学）；Department of Neonatology（新生儿科）；Shenzhen University of Advanced Technology General Hospital（深圳先进技术大学附属医院）；Department of Neurosurgery（神经外科）；The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；Department of Radiotherapy（放射治疗科）；University of Groningen, University Medical Center Groningen（格罗宁根大学，格罗宁根大学医学中心）；CSIRO Data61
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）
The Alberta Infant Motor Scale (AIMS) is a well-known assessment scheme that evaluates the gross motor development of infants by recording the number of specific poses achieved. With the aid of the image-based pose recognition model, the AIMS evaluation procedure can be shortened and automated, providing early diagnosis or indicator of potential developmental disorder. Due to limited public infant-related datasets, many works use the SMIL-based method to generate synthetic infant images for training. However, this domain mismatch between real and synthetic training samples often leads to perfo…
Assessing the alignment between infants' visual and linguistic experience using multimodal language models
Department of Psychology, Stanford University（心理学系，斯坦福大学）；Department of Psychology, University of California, San Diego（心理学系，加州大学圣地亚哥分校）；Department of Psychology, Tsinghua University（心理学系，清华大学）
Durham University（杜伦大学）；University of Glasgow（格拉斯哥大学）
Early prediction is clinically considered one of the essential parts of cerebral palsy (CP) treatment. We propose to implement a low-cost and interpretable classification system for supporting CP prediction based on General Movement Assessment (GMA). We design a Pytorch-based attention-informed graph convolutional network to early identify infants at risk of CP from skeletal data extracted from RGB videos. We also design a frequency-binning module for learning the CP movements in the frequency domain while filtering noise. Our system only requires consumer-grade RGB videos for training to supp…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Skull stripping for brain MR images is a basic segmentation task. Although many methods have been proposed, most of them focused mainly on the adult MR images. Skull stripping for infant MR images is more challenging due to the small size and dynamic intensity changes of brain tissues during the early ages. In this paper, we propose a novel CNN based framework to robustly extract brain region from infant MR image without any human assistance. Specifically, we propose a simplified but more robust flattened residual network architecture (FRnet). We also introduce a new boundary loss function to …
Shenzhen Automatic Rehabilitation Laboratory（深圳市自动康复实验室）；New York University（纽约大学）；Shenzhen Children’s Hospital（深圳市儿童医院）
Movement and pose assessment of newborns lets experienced pediatricians predict neurodevelopmental disorders, allowing early intervention for related diseases. However, most of the newest AI approaches for human pose estimation methods focus on adults, lacking publicly benchmark for infant pose estimation. In this paper, we fill this gap by proposing infant pose dataset and Deep Aggregation Vision Transformer for human pose estimation, which introduces a fast trained full transformer framework without using convolution operations to extract features in the early stages. It generalizes Transfor…
To make the earlier medical intervention of infants' cerebral palsy (CP), early diagnosis of brain damage is critical. Although general movements assessment(GMA) has shown promising results in early CP detection, it is laborious. Most existing works take videos as input to make fidgety movements(FMs) classification for the GMA automation. Those methods require a complete observation of videos and can not localize video frames containing normal FMs. Therefore we propose a novel approach named WO-GMA to perform FMs localization in the weakly supervised online setting. Infant body keypoints are f…
Object segmentation in infant's egocentric videos is a fundamental step in studying how children perceive objects in early stages of development. From the computer vision perspective, object segmentation in such videos pose quite a few challenges because the child's view is unfocused, often with large head movements, effecting in sudden changes in the child's point of view which leads to frequent change in object properties such as size, shape and illumination. In this paper, we develop a semi-automated, domain specific, method to address these concerns and facilitate the object annotation pro…
Northumbria University（诺森比亚大学）；NUTH NHS Foundation Trust（NUTH NHS基金会信托）
Providing early diagnosis of cerebral palsy (CP) is key to enhancing the developmental outcomes for those affected. Diagnostic tools such as the General Movements Assessment (GMA), have produced promising results in early diagnosis, however these manual methods can be laborious. In this paper, we propose a new framework for the automated classification of infant body movements, based upon the GMA, which unlike previous methods, also incorporates a visualization framework to aid with interpretability. Our proposed framework segments extracted features to detect the presence of Fidgety Movements…
LG Electronics（LG电子）；Seoul National University（首尔国立大学）
Human infants learn the names of objects and develop their own conceptual systems without explicit supervision. In this study, we propose methods for learning aligned vision-language conceptual systems inspired by infants' word learning mechanisms. The proposed model learns the associations of visual objects and words online and gradually constructs cross-modal relational graph networks. Additionally, we also propose an aligned cross-modal representation learning method that learns semantic representations of visual objects and words in a self-supervised manner based on the cross-modal relatio…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；University of California, Berkeley（加州大学伯克利分校）；Affiliated Eye Hospital of Nanchang University（南昌大学附属眼科医院）
Retinopathy of prematurity (ROP) is an abnormal blood vessel development in the retina of a prematurely-born infant or an infant with low birth weight. ROP is one of the leading causes for infant blindness globally. Early detection of ROP is critical to slow down and avert the progression to vision impairment caused by ROP. Yet there is limited awareness of ROP even among medical professionals. Consequently, dataset for ROP is limited if ever available, and is in general extremely imbalanced in terms of the ratio between negative images and positive ones. In this study, we formulate the proble…
LoCI-DiffCom: Longitudinal Consistency-Informed Diffusion Model for 3D Infant Brain Image Completion
School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trail Center（上海临床研究中心）
Aalto University（阿尔托大学）；Helsinki University Hospital（赫尔辛基大学医院）；University of Helsinki（赫尔辛基大学）；Tampere University（坦佩雷大学）；Scuola Superiore San’Anna（圣安娜高等学校）；IRCCS Fondazione Stella Maris（斯特拉·马里斯IRCCS基金会）；Kaasa solution GmbH（卡阿萨解决方案有限公司）
Infants' spontaneous and voluntary movements mirror developmental integrity of brain networks since they require coordinated activation of multiple sites in the central nervous system. Accordingly, early detection of infants with atypical motor development holds promise for recognizing those infants who are at risk for a wide range of neurodevelopmental disorders (e.g., cerebral palsy, autism spectrum disorders). Previously, novel wearable technology has shown promise for offering efficient, scalable and automated methods for movement assessment in adults. Here, we describe the development of …
Rewiring Development in Brain Segmentation: Leveraging Adult Brain Priors for Enhancing Infant MRI Segmentation
Department of Information Engineering University of Brescia (Italy)（信息工程系，布雷西亚大学）；School of Psychology & Neuroscience University of Glasgow (UK)（心理学与神经科学学院，格拉斯哥大学）；Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health University of Brescia (Italy)（医学与外科专科、放射科学与公共卫生系，布雷西亚大学）
Early diagnosis and intervention are clinically considered the paramount part of treating cerebral palsy (CP), so it is essential to design an efficient and interpretable automatic prediction system for CP. We highlight a significant difference between CP infants' frequency of human movement and that of the healthy group, which improves prediction performance. However, the existing deep learning-based methods did not use the frequency information of infants' movement for CP prediction. This paper proposes a frequency attention informed graph convolutional network and validates it on two consum…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）
During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mapped into a meaningful common space, allowing images to be retrieved using speech and vice versa. In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech. Concretely, we use an image-to-words multi-label visual classifier to tag images with soft textual labels, and then tra…
Predicting Infant Brain Connectivity with Federated Multi-Trajectory GNNs using Scarce Data
Imperial College London（伦敦帝国理工学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究中心）
Early Autism Diagnosis based on Path Signature and Siamese Unsupervised Feature Compressor
University of Pennsylvania（宾夕法尼亚大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The Affiliated Shenzhen School of Guangdong Experimental High School（广东实验中学附属深圳学校）；Pazhou Lab（琶洲实验室）
University Medical Center Göttingen（哥廷根大学医学中心）；Karolinska Institutet（卡罗林斯卡学院）；Medical University of Graz（格拉茨医科大学）；Leibniz-ScienceCampus Primate Cognition（莱布尼茨灵长类认知科学园区）；David Geffen UCLA School of Medicine（加州大学洛杉矶分校大卫·格芬医学院）；Region Stockholm（斯德哥尔摩地区）；Curtin University（科廷大学）；Victoria University of Wellington（惠灵顿维多利亚大学）
Video recording is a widely used method for documenting infant and child behaviours in research and clinical practice. Video data has rarely been shared due to ethical concerns of confidentiality, although the need of shared large-scaled datasets remains increasing. This demand is even more imperative when data-driven computer-based approaches are involved, such as screening tools to complement clinical assessments. To share data while abiding by privacy protection rules, a critical question arises whether efforts at data de-identification reduce data utility? We addressed this question by sho…
TU Kaiserslautern（凯泽斯劳滕工业大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
In this paper, we extend a symbolic association framework for being able to handle missing elements in multimodal sequences. The general scope of the work is the symbolic associations of object-word mappings as it happens in language development in infants. In other words, two different representations of the same abstract concepts can associate in both directions. This scenario has been long interested in Artificial Intelligence, Psychology, and Neuroscience. In this work, we extend a recent approach for multimodal sequences (visual and audio) to also cope with missing elements in one or both…
