视觉信息的世界模型可在给定动作序列的条件下预测未来图像帧。这种预测的场景演化由确定性变化(如机器人平台上的受控运动)和随机变化(如被操纵物体或先前遮挡区域的渲染)共同驱动。我们的核心见解是,这些随机变化的空间位置本身是可学习的。我们利用这一见解,联合回归确定性下一状态预测和空间不确定性图,再用该不确定性图指导条件扩散过程。我们提出了解耦漂移扩散世界模型(Decoupled Drift-Diffusion World Model,D3WM),该模型使用经过高斯负对数似然目标训练的Transformer,生成确定性下一状态预测和异方差随机不确定性的空间图。学习到的不确定性图为条件扩散模型提供了结构化空间门:确定性区域锚定到预测状态,而不确定区域(如遮挡恢复区域或动态纹理)则由扩散过程采样。我们在方差门控随机微分方程框架下构建了这种分解,将确定性漂移与随机扩散分离开来。D3WM在两种不同类型的动作空间(机器人桌面操纵和相机运动)上,相较于具有竞争力的基线模型展现出了更优的结果。
英文摘要:
World models of visual information predict future image frames conditioned on given action sequences. This predicted scene evolution is driven by deterministic changes such as controlled motion on a robot platform, as well as stochastic changes such as rendering of manipulated objects or previously occluded regions. Our key insight is that the spatial location of these stochastic changes is itself learnable. We leverage this insight by jointly regressing both the deterministic next-state prediction and the spatial uncertainty map, then using the uncertainty map to guide a conditional diffusion process. We propose Decoupled Drift-Diffusion World Model (D3WM), which uses a transformer trained with a Gaussian negative-log-likelihood objective to produce both the deterministic next-state prediction and a spatial map of heteroscedastic aleatoric uncertainty. The learned uncertainty map then provides a structured spatial gate for a conditional diffusion model: deterministic regions are anchored to the predicted state, while uncertain regions, such as disocclusions or dynamic textures, are sampled by the diffusion process. We frame this decomposition under a variance-gated stochastic differential equation that separates deterministic drift from stochastic diffusion. D3WM shows improved results over competitive baselines in two distinct types of action spaces: robotic tabletop manipulation and camera motion.
DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion
College of Computer Science;Technology, Zhejiang University, Hangzhou, China;School of Software Technology \& DUT-RU International School of ISE, Dalian University of Technology, Dalian, China;College of Information Science;Technology, Dalian Maritime University, Dalian, China
Decoupled Video Generation with Chain of Training-free Diffusion Model Experts
University of Sydney(悉尼大学);Southeast University(东南大学);Central South University(中南大学);Shanghai Jiaotong University(上海交通大学);Sensetime Research(商汤科技研究院);Hong Kong University of Science and Technology(香港科技大学)
Decouple, Reorganize, and Fuse: A Multimodal Framework for Cancer Survival Prediction
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);State Key Laboratory of Transvascular Implantation Devices of the Second Affiliated Hospital, Zhejiang University School of Medicine(浙江大学医学院第二附属医院血管植入设备国家重点实验室);Transvascular Implantation Devices Research Institute(血管植入设备研究院);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院)
Decoupled Spatial-Temporal Transformer for Video Inpainting
CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong(香港中文大学-商汤科技联合实验室,香港中文大学);SenseTime Research(商汤科技研究院);Zhejiang University(浙江大学);School of CST, Xidian University(西安电子科技大学计算机科学与技术学院)
D$^2$Turb: Depth-Aware Simulation and Decoupled Learning for Single-Frame Atmospheric Turbulence Mitigation
University of Electronic Science and Technology of China(电子科技大学);Beijing Institute of Space Mechanics and Electricity(北京空间机械与电子研究所);School of Physics, Northwest University Xi'an(西安西北大学物理学院)
Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition
School of Computer Science and Engineering, Nanjing University of Science and Technology(计算机科学与工程学院,南京理工大学);State Key Lab of Brain-Machine Intelligence, Zhejiang University(脑机智能国家重点实验室,浙江大学)
DriftGuard: A Hierarchical Framework for Concept Drift Detection and Remediation in Supply Chain Forecasting
Department of Computer Science Engineering, Muffakham Jah College of Engineering and Technology(穆法卡姆·贾赫工程与技术学院计算机科学工程系);Department of CSE, Shadan Women's College of Engineering and Technology(沙达恩女子工程与技术学院计算机科学与工程系)