发表机构
HKUST(GZ); ATH, Alibaba（香港科技大学（广州）; 阿里巴巴 ATH 实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ReWorld通过训练分离、推理约束解决交互式世界模型的控制与记忆矛盾，结合混合注意力等技术，在三轴评估中优于6种模型，实现长程视频生成与记忆。
AI 中文摘要
交互式世界模型必须跟随用户的动作、记住其展示过的场景并实时生成内容，二者之间存在结构性矛盾：控制需要短程视野，而记忆需要无界视野。ReWorld在训练阶段将二者分离，在推理阶段对其进行约束。混合头注意力窗口将大多数注意力头限制在近期历史，同时一小部分全局注意力头会关注整个历史；随机头路由避免任何能力绑定到特定注意力头；随机块丢弃使稀疏历史符合分布。在推理阶段，全部历史在固定预算下运行：由姿态索引的地标库支持的受限KV缓存，模型从中检索当前姿态最近的地标。度量尺度对齐的数据引擎将8种来源——虚幻引擎渲染的飞行视频、游戏漫游和真实世界素材——置于同一物理动作尺度，使得相同按键在所有来源中移动相机的距离一致，而回文轨迹提供了记忆训练所需的重访证据。仅在LoRA适配器上进行分布匹配蒸馏将采样压缩至4步：一个主干网络同时支持高保真多步模式和实时交互模式，生成704×1280视频，涵盖 photorealistic、游戏风格和风格化世界。在涵盖动作跟随、长程记忆和视频质量的三轴协议下，与6种最新交互式世界模型相比，ReWorld实现了最佳控制保真度（旋转误差11.95°，且相机运动一致性最佳）和最佳生成质量；在长达数分钟的往返滚动（64秒，384个隐变量）中，其固定12块缓存仍能再生起始视图——而在该滚动长度下，滑动窗口早已将证据逐出，全KV注意力也会耗尽内存。
英文摘要
An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ($11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ($64$\,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.
Comments21 pages, 9 figures. Project page: https://zhifeichen097.github.io/ReWorld/
