LiveLight：具备交互式控制的实时流视频重光照
LiveLight: Real-time Streaming Video Relighting with Interactive Control
浏览论文内容
中文总结 AI 辅助
本研究提出首个基于扩散模型的实时流视频重光照框架LiveLight，通过三项关键设计解决三大挑战，在实时速度下达到最优重光照质量，将公开发布相关资源以推动该领域研究。
中文摘要 AI 辅助
我们提出LiveLight，这是首个基于扩散模型的、支持交互式3D光照控制的实时流视频重光照框架。实现这一目标并非易事，因为需要克服三个关键挑战：将动态3D光照有效注入扩散模型、在极低的NFE（函数评估次数）预算下维持高保真生成以实现实时速度，以及为交互式控制提供连续流支持。为解决这些痛点，我们提出三项关键设计：其一，为实现精准光照注入，我们设计了一种轻量适配器，将多平面光照辐照度（MPLI）条件——编码3D光照几何的深度感知辐照度图——直接输入扩散骨干网络；其二，为避免低NFE下的渲染质量下降以实现实时蒸馏，我们引入了几何引导反馈分支，该训练时约束利用冻结的几何估计器，强制实现与深度和法向一致的重光照，确保几何上合理的着色，且不增加推理开销；其三，为支持流式交互，我们开发了渐进式滚动窗口策略，该策略维护不同噪声水平的潜在块去噪阶梯，通过传播中间状态，保证时间一致性，并支持逐帧参考刷新的任意长视频重光照。在真实世界和合成基准上的大量实验表明，LiveLight达到了最先进的重光照质量，同时实现了实时速度，在时间稳定性、光照可控性和用户偏好方面显著优于离线基线。为推动实时交互式重光照研究，我们将公开发布模型、训练数据和合成数据生成器。
英文摘要
We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.
发表机构
- HKUST（香港科技大学）
- University of Macau（澳门大学）
- THU（清华大学）
- UoT（多伦多大学）
- University of Tuebingen（蒂宾根大学）
机构由 AI 辅助整理，请以论文原文为准。
