Heterogeneity in sensors and actuators across environments poses a significant challenge to building large-scale pre-trained world models on top of this low-dimensional sensor information. In this work, we explore pre-training world models for heterogeneous environments by addressing key transfer barriers in both data diversity and model flexibility. We introduce UniTraj, a unified dataset comprising over one million trajectories from 80 environments, designed to scale data while preserving critical diversity. Additionally, we propose TrajWorld, a novel architecture capable of flexibly handling varying sensor and actuator information and capturing environment dynamics in-context. Pre-training TrajWorld on UniTraj yields substantial gains in transition prediction, achieves a new state-of-the-art for off-policy evaluation, and also delivers superior online performance of model predictive control. To the best of our knowledge, this work, for the first time, demonstrates the transfer benefits of world models across heterogeneous and complex control environments. Code and data are available at https://github.com/thuml/TrajWorld.
Trajectory prediction for heterogeneous agents: A performance analysis on small and imbalanced datasets
Robert Bosch GmbH, Corporate Research(罗伯特·博世有限公司企业研究部);Finnish Center for Artificial Intelligence (FCAI)(芬兰人工智能中心);Technical University of Munich(慕尼黑工业大学)
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University(上海交通大学并行与分布式系统研究所(IPADS));Department of Precision Instrument, Tsinghua University(清华大学精密仪器系);National Innovation Institute of High-end Smart Appliances(国家高端智能家电创新研究院)
EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction
Department of Civil, Architectural, and Environmental Engineering, The University of Texas at Austin(土木、建筑与环境工程系,德克萨斯大学奥斯汀分校);Meta Reality Labs(Meta现实实验室);School of Architecture, The University of Texas at Austin(建筑学院,德克萨斯大学奥斯汀分校)
Discovering Heterogeneous Subsequences for Trajectory Classification
Federal Institute of Santa Catarina(圣卡塔琳娜联邦研究所);Universidade Federal de Santa Catarina(圣卡塔琳娜联邦大学);Federal University for Latin American Integration(拉丁美洲一体化联邦大学)
Fairness-Aware Federated Learning with Trajectory Shapley Value
Faculty of Mathematics, Ecole Normale Supérieure Paris-Saclay(巴黎-萨克雷大学数学系);Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship, Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔兰根-纽伦堡弗里德里希-亚历山大大学数学系)
MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院);Bio-X Institutes, Key Laboratory for the Genetics of Developmental and Neuropsychiatric Disorders, Shanghai Jiao Tong University(上海交通大学Bio-X研究院、发育与神经精神疾病遗传学重点实验室);Shanghai Key Laboratory of Psychotic Disorders, Brain Science and Technology Research Center, Shanghai Jiao Tong University(上海精神疾病重点实验室、脑科学与技术研究中心,上海交通大学)
Heterogeneous Graph-based Trajectory Prediction using Local Map Context and Social Interactions
FZI Research Center for Information Technology(FZI信息技术研究中心);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Bosch Center for Artificial Intelligence(博世人工智能中心)
Heterogeneous Graph Contrastive Learning with Spectral Augmentation
School of Cyber Science and Engineering, Southeast University(东南大学网络空间安全学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systems
Georgia Institute of Technology(佐治亚理工学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
UniTraj: Learning a Universal Trajectory Foundation Model from Billion-Scale Worldwide Traces
Southern University of Science and Technology(南方科技大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System
Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School(药理流行病学与药效学部,医学部,布里奇沃特医院,哈佛医学院);Harvard T.H. Chan School of Public Health, Harvard University(哈佛大学T.H. 陈公共卫生学院);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(自然与人工智能研究学院,哈佛大学);Broad Institute of MIT and Harvard, Cambridge, MA, USA(MIT与哈佛大学Broad研究所,剑桥,马萨诸塞州,美国);Harvard Data Science Initiative, Harvard University(哈佛大学数据科学计划)
Understanding Trajectory Heterogeneity in Federated World Model Learning
Wuhan University(武汉大学);China Academy of Information and Communications Technology(中国信息通信研究院);The University of Sydney(悉尼大学);Tsinghua University(清华大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Hamiltonian Graph Inference Networks: Joint structure discovery and dynamics prediction for lattice Hamiltonian systems from trajectory data
School of Mathematics and Statistics(数学与统计学学院);University of Massachusetts(马萨诸塞大学);Seoul National University(首尔国立大学);Northeast Normal University(东北师范大学);Great Bay University(大贝大学)