For robots to perform assistive tasks in unstructured home environments, they must learn and reason on the semantic knowledge of the environments. Despite a resurgence in the development of semantic reasoning architectures, these methods assume that all the training data is available a priori. However, each user's environment is unique and can continue to change over time, which makes these methods unsuitable for personalized home service robots. Although research in continual learning develops methods that can learn and adapt over time, most of these methods are tested in the narrow context of object classification on static image datasets. In this paper, we combine ideas from continual learning, semantic reasoning, and interactive machine learning literature and develop a novel interactive continual learning architecture for continual learning of semantic knowledge in a home environment through human-robot interaction. The architecture builds on core cognitive principles of learning and memory for efficient and real-time learning of new knowledge from humans. We integrate our architecture with a physical mobile manipulator robot and perform extensive system evaluations in a laboratory environment over two months. Our results demonstrate the effectiveness of our architecture to allow a physical robot to continually adapt to the changes in the environment from limited data provided by the users (experimenters), and use the learned knowledge to perform object fetching tasks.
University of Southern California, Viterbi School of Engineering（南加州大学维特比工程学院）
Socially assistive robots (SAR) have shown great potential to augment the social and educational development of children with autism spectrum disorders (ASD). As SAR continues to substantiate itself as an effective enhancement to human intervention, researchers have sought to study its longitudinal impacts in real-world environments, including the home. Computational personalization stands out as a central computational challenge as it is necessary to enable SAR systems to adapt to each child's unique and changing needs. Toward that end, we formalized personalization as a hierarchical human ro…
SPICED: A Synaptic Homeostasis-Inspired Framework for Unsupervised Continual EEG Decoding
State Key Laboratory of Brain-machine Intelligence（脑机智能国家重点实验室）；College of Computer Science and Technology（计算机科学与技术学院）；Department of Neurobiology, Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital（神经生物学系、附属心理健康中心及杭州市第七人民医院）；MOE Frontier Science Center for Brain Science and Brain-machine Integration（教育部脑科学与脑机整合前沿科学中心）
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science Peking University（北京大学计算机学院）；AIRobotics
CoMetaPNS: Continually Meta-learning Personalized Neural Surrogates for Cardiac Electrophysiology Simulations
Golisano College of Computing and Information Sciences, Rochester Institute of Technology（罗切斯特理工学院戈利萨诺计算与信息科学学院）；Department of Computer Science, Rowan University（罗文大学计算机科学系）；The University of Utah（犹他大学）
This work investigates the task of unsupervised model personalization, adapted to continually evolving, unlabeled local user images. We consider the practical scenario where a high capacity server interacts with a myriad of resource-limited edge devices, imposing strong requirements on scalability and local data privacy. We aim to address this challenge within the continual learning paradigm and provide a novel Dual User-Adaptation framework (DUA) to explore the problem. This framework flexibly disentangles user-adaptation into model personalization on the server and local data regularization …
University of Bologna（博洛尼亚大学）；University of Pisa（比萨大学）
On-device training for personalized learning is a challenging research problem. Being able to quickly adapt deep prediction models at the edge is necessary to better suit personal user needs. However, adaptation on the edge poses some questions on both the efficiency and sustainability of the learning process and on the ability to work under shifting data distributions. Indeed, naively fine-tuning a prediction model only on the newly available data results in catastrophic forgetting, a sudden erasure of previously acquired knowledge. In this paper, we detail the implementation and deployment o…
Rochester Institute of Technology（罗切斯特理工学院）；University of Rochester（罗切斯特大学）
Real-time on-device continual learning is needed for new applications such as home robots, user personalization on smartphones, and augmented/virtual reality headsets. However, this setting poses unique challenges: embedded devices have limited memory and compute capacity and conventional machine learning models suffer from catastrophic forgetting when updated on non-stationary data streams. While several online continual learning models have been developed, their effectiveness for embedded applications has not been rigorously studied. In this paper, we first identify criteria that online cont…
HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs
Institut Curie, Université Paris-Saclay（巴黎-萨克勒大学Curie研究所）；Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University（索邦大学智能系统与机器人研究所）；Assistance Publique – Hôpitaux de Paris (AP-HP), Université Paris Cité（巴黎公共医院（AP-HP）与巴黎城市大学）
Continual Learning: Applications and the Road Forward
KU Leuven（鲁汶大学）；Toyota Motor Europe（丰田汽车欧洲公司）；University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；University of Tübingen（蒂宾根大学）；University of Pisa（比萨大学）；University of Applied Sciences Fulda（富尔达应用科学大学）；NAVER LABS Europe（NAVER欧洲实验室）；University of Munich (LMU)（慕尼黑大学）；University of Rochester（罗切斯特大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；TU Darmstadt（达姆施塔特工业大学）；Google DeepMind（谷歌DeepMind）；Université Paris-Saclay（巴黎萨克雷大学）；CEA LIST（法国原子能和替代能源委员会 LIST 研究所）；Baylor College of Medicine（贝勒医学院）；Computer Vision Center UAB（巴塞罗那自治大学计算机视觉中心）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science（计算机学院）；Peking University（北京大学）；CUHK（香港中文大学）；University of Science Technology of China（中国科学技术大学）
Sensor-based human activity recognition (HAR), i.e., the ability to discover human daily activity patterns from wearable or embedded sensors, is a key enabler for many real-world applications in smart homes, personal healthcare, and urban planning. However, with an increasing number of applications being deployed, an important question arises: how can a HAR system autonomously learn new activities over a long period of time without being re-engineered from scratch? This problem is known as continual learning and has been particularly popular in the domain of computer vision, where several tech…
Position: Modular Memory is the Key to Continual Learning Agents
University of Bremen（不莱梅大学）；Seoul National University（首尔国立大学）；Computer Vision Center Barcelona（巴塞罗那计算机视觉中心）；University of Florence（佛罗伦萨大学）；Microsoft Research（微软研究院）；HEC Montreal, Mila--Quebec AI Institute, Canada CIFAR AI Chair（蒙特利尔HEC学院、魁北克人工智能研究所、加拿大CIFAR人工智能 chair）；Bielefeld University（比勒海姆大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Rochester（罗切斯特大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Nankai University（南开大学）；LUISS University（卢西亚诺大学）；Stony Brook University（石溪大学）；University of Tübingen（图宾根大学）；University of Trento, FBK（特伦托大学，FBK）；Tsinghua University（清华大学）；University of Groningen（Groningen大学）
EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能国家重点实验室）；University of Hong Kong（香港大学）
Personalized Federated Continual Learning via Multi-granularity Prompt
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）；Webank（微众银行）；College of Computer Science, Sichuan University（四川大学计算机学院）；JD Intelligent Cities Research（京东城市智能研究院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）
University of Southern California（南加州大学）；Massachusetts Institute of Technology（麻省理工学院）
Affect-aware socially assistive robotics (SAR) has shown great potential for augmenting interventions for children with autism spectrum disorders (ASD). However, current SAR cannot yet perceive the unique and diverse set of atypical cognitive-affective behaviors from children with ASD in an automatic and personalized fashion in long-term (multi-session) real-world interactions. To bridge this gap, this work designed and validated personalized models of arousal and valence for children with ASD using a multi-session in-home dataset of SAR interventions. By training machine learning (ML) algorit…
