Scaling up deep Reinforcement Learning (RL) methods presents a significant challenge. Following developments in generative modelling, model-based RL positions itself as a strong contender. Recent advances in sequence modelling have led to effective transformer-based world models, albeit at the price of heavy computations due to the long sequences of tokens required to accurately simulate environments. In this work, we propose $Δ$-IRIS, a new agent with a world model architecture composed of a discrete autoencoder that encodes stochastic deltas between time steps and an autoregressive transformer that predicts future deltas by summarizing the current state of the world with continuous tokens. In the Crafter benchmark, $Δ$-IRIS sets a new state of the art at multiple frame budgets, while being an order of magnitude faster to train than previous attention-based approaches. We release our code and models at https://github.com/vmicheli/delta-iris.
Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
Broad Institute of MIT and Harvard, Cambridge, MA, USA(麻省理工学院与哈佛大学Broad研究所);Massachusetts Institute of Technology, Cambridge, MA, USA(麻省理工学院);Koch Institute for Integrative Cancer Research, MIT, Cambridge, MA, USA(麻省理工学院Koch整合癌症研究 institute);Department of Neurology and Neurosurgery, McGill University, Montreal, Canada(麦吉尔大学神经学与神经外科系);The Montreal Neurological Hospital-Institute, Montreal, Canada(蒙特利尔神经科学医院-研究所)
$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家级重点实验室);Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人工智能与机器人学院,西安交通大学)
Phaedra: Learning High-Fidelity Discrete Tokenization for the Physical Science
ETH AI Center(苏黎世联邦理工学院人工智能中心);IBM Research Europe(IBM欧洲研究院);Seminar for Applied Mathematics, ETH Zurich(苏黎世联邦理工学院应用数学系);Swiss Data Science Center, ETH Zurich(瑞士数据科学中心,苏黎世联邦理工学院)
RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究所);Robbyant, Ant Group(蚂蚁集团 Robbyant);Hongkong University of Science and Technology(香港科技大学)
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
Tsinghua University(清华大学);Westlake University(西湖大学);Zhejiang University(浙江大学);Hong Kong University of Science and Technology(香港科学与技术大学);OPPO;Shanghai AI Lab(上海人工智能实验室)
Network-Aware Bilinear Tokenization for Brain Functional Connectivity Representation Learning
Department of Radiology, Weill Cornell Medicine, New York, NY, USA.(韦尔·科恩医学中心放射科, 纽约, NY, 美国);School of Electrical and Computer Engineering, Cornell University and Cornell Tech, New York, NY, USA.(康奈尔大学电气与计算机工程学院及康奈尔科技, 纽约, NY, 美国)
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Yuanpei College, Peking University(北京大学元培学院);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院);Google Research(谷歌研究);Tel-Aviv University(特拉维夫大学)