In this paper, we propose Multi-View Dreaming, a novel reinforcement learning agent for integrated recognition and control from multi-view observations by extending Dreaming. Most current reinforcement learning method assumes a single-view observation space, and this imposes limitations on the observed data, such as lack of spatial information and occlusions. This makes obtaining ideal observational information from the environment difficult and is a bottleneck for real-world robotics applications. In this paper, we use contrastive learning to train a shared latent space between different viewpoints, and show how the Products of Experts approach can be used to integrate and control the probability distributions of latent states for multiple viewpoints. We also propose Multi-View DreamingV2, a variant of Multi-View Dreaming that uses a categorical distribution to model the latent state instead of the Gaussian distribution. Experiments show that the proposed method outperforms simple extensions of existing methods in a realistic robot control task.
Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis
University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Henan University(河南大学)
A Learnable Multi-views Contrastive Framework with Reconstruction Discrepancy for Medical Time-Series
School of Medicine, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)医学院);Digital Department, China Unicom Global Co., Ltd(中国联通全球有限公司数字部);University of the Chinese Academy of Sciences(中国科学院大学);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training
Wuhan University(武汉大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Huazhong University of Science and Technology(华中科技大学);Monash University(莫纳什大学);Yunnan University(云南大学)
CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Peng Cheng Laboratory(鹏城实验室);Migu Culture Technology Co., Ltd(咪咕文化科技有限公司);School of Computer Science, University of Birmingham(伯明翰大学计算机学院);School of Arts, Peking University(北京大学艺术学院)
Contrastive Continual Multi-view Clustering with Filtered Structural Fusion
School of Computer, National University of Defense Technology(国防科技大学计算机学院);School of Computer Science and Technology, Harbin University of Science and Technology(哈尔滨理工大学计算机科学与技术学院)
Pathology-Aware Multi-View Contrastive Learning for Patient-Independent ECG Reconstruction
Department of Computer Science and Engineering, Indian Institute of Technology Roorkee(计算机科学与工程系,印度理工学院Roorkee);Department of Biosciences and Bioengineering, Indian Institute of Technology Roorkee(生物科学与生物工程系,印度理工学院Roorkee)
DGCPath: Distribution-Aware Generative Contrastive Framework for Self-supervised Path Representation Learning -- Extended Version
Aalborg University(奥尔堡大学);The Hong Kong Polytechnic University(香港理工大学);Chongqing Univeristy of Posts and Telecommunications(重庆邮电大学);East China Normal University(华东师范大学)
SMART: Semantic Matching Contrastive Learning for Partially View-Aligned Clustering
College of Computer Science and Technology, Huaqiao University(华侨大学计算机科学与技术学院);Department of Computer Science, Shantou University(汕头大学计算机科学系);Chongqing Key Laboratory of Nonlinear Circuits and Intelligent Information Processing, College of Electronic and Information Engineering, Southwest University(西南大学电子信息工程学院非线性电路与智能信息处理重点实验室);School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledge
Xidian University(西安电子科技大学);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院);Ruijin Hospital(瑞金医院);Shanghai Institute of Traumatology and Orthopedics(上海市伤骨科研究所);Harbin Medical University(哈尔滨医科大学);The First Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第一医院);Brown University(布朗大学)
A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanyang Technological University(南洋理工大学);Dong Fang Hospital, Beijing University of Chinese Medicine(北京中医药大学东方医院)
Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University(西安交通大学智能网络与网络安全教育部重点实验室);School of Artificial Intelligence, Xi'an Jiaotong University(西安交通大学人工智能学院)
Uncovering Capabilities of Model Pruning in Graph Contrastive Learning
State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂与关键软件环境国家重点实验室);School of Statistics and Mathematics, Central University of Finance and Economics(中央财经大学统计与数学学院)
University of Electronic Science and Technology of China(电子科技大学);Tencent Inc.(腾讯公司);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng National Lab(鹏城国家实验室)