Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform. Similar to human infants, artificial agents can learn language while interacting with their environment. In this work, first, we present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Building on our previous Paired Variational Autoencoders (PVAE) model, we demonstrate the superiority of the variational autoencoder over standard autoencoders by experimenting with cubes of different colours, and by enabling the production of alternative vocabularies. Additional experiments show that the model's channel-separated visual feature extraction module can cope with objects of different shapes. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model, i.e., Bidirectional Encoder Representations from Transformers (BERT), enabling the model to go beyond comprehending only the predefined descriptions that the network has been trained on; the recognition of action descriptions generalises to unconstrained natural language as the model becomes capable of understanding unlimited variations of the same descriptions. Our experiments suggest that using a pretrained language model as the language encoder allows our approach to scale up for real-world scenarios with instructions from human users.
Neural Style Transfer with Twin-Delayed DDPG for Shared Control of Robotic Manipulators
CNRS（法国国家科学研究中心）；Univ Rennes（雷恩大学）；Inria（法国国家信息与自动化研究所）；IRISA（雷恩高等计算机与通信科学研究院）；Universidad Carlos III de Madrid (UC3M)（马德里卡洛斯三世大学）；Robotics Lab（机器人实验室）
Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；X-Humanoid Robots（X-Humanoid机器人）；The University of Hong Kong（香港大学）；Tsinghua University（清华大学）
GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
Tsinghua University（清华大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）
