Sensorimotor World Models: Perception for Action via Inverse Dynamics
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Department of Computer Science, Brown University(布朗大学计算机科学系);ELLIS Institute(ELLIS研究所);ETH Zürich(苏黎世联邦理工学院)
Petr Ivashkov, Randall Balestriero, Bernhard Schölkopf
Perception for action suggests that representations of the world should be shaped not by visual fidelity alone, but by their relevance for actions. At the same time, latent JEPA-style world models advocate learning compact predictive states from high-dimensional observations to facilitate the prediction of future states, but end-to-end training of these models is nontrivial because representations may collapse if our only goal is to construct a latent state that is easy to predict. We introduce a sensorimotor world model (SMWM): a latent world model trained end-to-end with inverse dynamics regularization. This single regularizer addresses both issues: it prevents representation collapse and induces action-aligned representations. By forcing latent states to preserve information about the action underlying a transition, it biases the model toward the controllable degrees of freedom of the environment while discarding uncontrollable distractors. This yields stable latent world models trained from offline, reward-free trajectories, without frozen encoders, exponential moving averages, or complex latent regularizers. Empirically, SMWM learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
The Rate-Distortion-Perception-Classification Tradeoff: Joint Source Coding and Modulation via Inverse-Domain GANs
School of Informatics, Xiamen University(厦门大学信息学院);School of Electronics and Information Engineering, Guangxi Normal University(广西师范大学电子与信息工程学院);School of Engineering & Physical Sciences, Heriot-Watt University(赫瑞瓦特大学工程与物理科学学院)