基于物理信息的视频生成:通过混合专家潜在对齐
Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment
AI总结:
提出PILA框架,通过混合专家潜在对齐将物理结构化潜在引导注入预训练视频模型的冻结流匹配动力学,以提升生成视频的物理合理性。
摘要:
大规模视频生成模型在语义一致性和视觉质量方面取得了显著进展,生成的视频越来越连贯且视觉上令人信服。然而,由像素级拟合引发的动态过程自然无法适应支配真实世界运动和交互的规律性,导致在物理合理性方面持续存在不足。为解决这一局限,我们提出了PILA(物理信息潜在对齐),一个将物理结构化的潜在引导注入预训练视频模型冻结流匹配动力学的框架。具体而言,PILA首先采用锚定场估计,将冻结生成器的潜在变量映射到一个由场代理槽组织的可操作物理属性库中,利用可观测运动作为运动学锚点来构建较难直接观测的代理。为处理真实世界动态的异质性,PILA采用基于物理类别的混合专家设计。标签先验掩码专家路由选择特定类别的算子专家,其精炼结果通过从物理关系中抽象出的操作残差进行正则化。最后,精炼后的代理被融合回物理属性库,并解码为流匹配向量场的修正,从而在保持预训练骨干网络视觉先验的同时注入物理感知引导。通过在Wan 2.1-1.3B上进行分阶段适配器训练,并将学到的适配器直接迁移到Wan 2.2-14B,PILA在VBench-2.0、VideoPhy-2和PhyGenBench上,在视觉质量和基准测量的物理合理性方面均达到了最先进的结果。
英文摘要:
Large-scale video generation models have made remarkable progress in semantic consistency and visual quality, producing videos that are increasingly coherent and visually convincing. Nevertheless, the dynamics induced by pixel-level fitting do not naturally accommodate the regularities that govern real-world motion and interaction, resulting in persistent shortcomings in physical plausibility. To address this limitation, we propose \textbf{PILA} (Physics-Informed Latent Alignment), a framework that injects physics-structured latent guidance into the frozen flow-matching dynamics of pretrained video models. Specifically, PILA first employs anchored field estimation to map frozen-generator latents into an operational physical attribute bank organized by field-proxy slots, using observable motion as a kinematic anchor for constructing less directly observed proxies. To handle the heterogeneity of real-world dynamics, PILA adopts a mixture-of-experts design over physical categories. Label-prior masked expert routing selects category-specific operator experts, whose refinements are regularized by operational residuals abstracted from physical relations. Finally, the refined proxies are fused into the physical attribute bank and decoded into a correction to the flow-matching vector field, injecting physics-aware guidance while preserving the visual prior of the pretrained backbone. With staged adapter training on Wan 2.1-1.3B and direct transfer of the learned adapter to Wan 2.2-14B, PILA achieves state-of-the-art results on VBench-2.0, VideoPhy-2, and PhyGenBench in both visual quality and benchmark-measured physical plausibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。