Learning efficient representations for decision-making policies is a challenge in imitation learning (IL). Current IL methods require expert demonstrations, which are expensive to collect. Additionally, they are not explicitly trained to understand the environment. Consequently, they have underdeveloped world models. Self-supervised learning (SSL) offers an alternative, as it can learn a world model from diverse, unlabeled data. However, most SSL methods are inefficient because they operate in raw input space. In this work, we propose ACT-JEPA, a novel architecture that unifies IL and SSL to enhance policy representations. It is trained end-to-end to jointly predict 1) action sequences and 2) latent observation sequences. To learn in latent space, we utilize Joint-Embedding Predictive Architecture, which allows the model to filter out irrelevant details and learn a robust world model. We evaluate ACT-JEPA in different environments and across multiple tasks. Our results show that it outperforms the strongest baseline in all environments. ACT-JEPA achieves up to 40% improvement in world model understanding and up to 10% higher task success rate. Finally, we show that predicting latent observation sequences effectively generalizes to predicting action sequences. This work demonstrates how integrating IL and SSL leads to efficient policy representation learning, an improved world model, and a higher task success rate.
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
Biomedical AI Research Lab, University of California, Los Angeles, USA(生物医学人工智能研究实验室,加州大学洛杉矶分校);Department of Bioengineering, University of California, Los Angeles, USA(生物工程系,加州大学洛杉矶分校);Medical Informatics Home Area, University of California, Los Angeles, USA(医学信息学家庭区域,加州大学洛杉矶分校);Department of Electrical and Computer Engineering, University of California, Los Angeles, USA(电气与计算机工程系,加州大学洛杉矶分校);Department of Pathology, University of California, Los Angeles, USA(病理学系,加州大学洛杉矶分校);Department of Radiology, University of California, Los Angeles, USA(放射学系,加州大学洛杉矶分校)
MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
Meta AI;Inria(法国国家信息与自动化研究所);École normale supérieure(巴黎高等师范学院);CNRS(法国国家科学研究中心);PSL Research University(巴黎文理研究大学);Courant Institute, New York University(纽约大学柯朗研究所);Center for Data Science, New York University(纽约大学数据科学中心)
From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
Department of Information and Communication Technology, School of Technology, Pandit Deendayal Energy University, Gandhinagar, Gujarat 382007, India(信息与通信技术系,技术学院,潘迪·德恩达尔能源大学,古吉拉特邦加尔各纳,382007,印度)
JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics
GREYC, Normandy University, Unicaen, ENSICAEN, UMR CNRS 6072(GREYC,诺曼底大学,Unicaen,ENSICAEN,CNRS UMR 6072);LPC, Normandy University, Unicaen, ENSICAEN, IN2P3, UMR CNRS 6534(LPC,诺曼底大学,Unicaen,ENSICAEN,IN2P3,CNRS UMR 6534)
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
Carnegie Mellon University(卡内基梅隆大学);Amazon GenAI(亚马逊生成人工智能);James Silberrad Brown Center for Artificial Intelligence(詹姆斯·西伯拉德·布朗人工智能中心);University of Bristol(布里斯托大学);Stanford University(斯坦福大学);Northeastern University(东北大学);New York University(纽约大学)
Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Computer Science and Technology, Harbin Institute of Technology, Weihai(哈尔滨工业大学(威海)计算机科学与技术学院);Faculty of Computing, Harbin Institute of Technology, Harbin(哈尔滨工业大学计算学部);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning
Graduate School of Artificial Intelligence Convergence Engineering, Changwon National University(昌原国立大学人工智能融合工程研究生院);Department of Artificial Intelligence Engineering, Changwon National University(昌原国立大学人工智能工程系)