并非所有动作都同等重要：重新思考灵巧世界模型的条件化
Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model
AI总结：
针对高自由度灵巧动作中不同分量重要性不均导致优化失衡的问题，提出DexAC-WM，通过动作标记化保留维度级语义，结合局部细化与全局调制对齐动作与视觉动态，并引入语义分支提供物体场景先验，显著提升视频预测的真实感和动作一致性。
摘要：
近期基于动作条件的世界模型在建模复杂交互和预测未来状态方面取得了显著进展。尽管这些模型通常由更强的视觉表示和模型容量驱动，但动作条件化本身仍未得到充分探索。现有方法大多将整个动作序列压缩为单一表示，这适用于低自由度控制，但在高自由度场景中可靠性下降。我们观察到高自由度灵巧动作本质上是异质的，跨越多个数量级，大规模运动与细微但重要的信号共存。当统一聚合时，优化在动作分量间表现出不平衡，阻碍了细粒度效应的建模并影响动作保真度。因此，我们提出DexAC-WM，将动作条件化视为结构化过程而非全局压缩。DexAC通过动作标记化保留维度级语义，并通过局部细化与全局调制将动作信号与视觉动态对齐。为解决现有世界模型中高级语义基础有限的问题，我们进一步引入语义分支，提供丰富的物体场景先验，使世界模型能够捕捉动态视觉细节，同时支持高自由度动作条件视频预测。在EgoDex和EgoVerse上的实验表明，将语义分支与DexAC结合显著改善了FID、FVD和PCK，在视觉时间真实性和动作跟随一致性上均有提升。我们进一步验证了DexAC可扩展到其他骨干网络，展示了结构化动作条件化设计的可扩展性。这些结果表明，将世界模型扩展到高自由度控制需要结构化的动作建模和语义基础。
英文摘要：
Recent advances in action-conditioned world models show promising progress in modeling complex interactions and forecasting future states under diverse action sequences. While these models are often driven by stronger visual representations and model capacity, action conditioning itself remains underexplored. Most existing approaches compress the entire action sequence into a single representation, which works well for low-DoF control but becomes less reliable in high-DoF scenarios. We observe that high-DoF dexterous actions are inherently heterogeneous, spanning multiple orders of magnitude, where large-scale motions coexist with subtle but important signals. When uniformly aggregated, optimization exhibits an imbalance across action components, which hinders the modeling of fine-grained effects and affects action fidelity. We therefore propose DexAC-WM, which treats action conditioning as a structured process rather than global compression. DexAC preserves dimension-level semantics via action tokenization and aligns action signals with visual dynamics through local refinement and global modulation. To address the limited high-level semantic grounding in existing world models, we further introduce a semantic branch that provides rich object-scene priors, which enables world model to capture dynamic visual details while supporting high-DoF action-conditioned video prediction. Experiments on EgoDex and EgoVerse show that combining the semantic branch with DexAC significantly improves FID, FVD, and PCK, demonstrating gains in visual-temporal realism and action-following consistency. We further verify that DexAC extends to other backbones, showing the scalability of our structured action-conditioning design. These results suggest that scaling world models to high-DoF control requires both structured action modeling and semantic grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
