Morphe: 基于视觉基础模型的高保真生成视频流媒体
Morphe: High-Fidelity Generative Video Streaming with Vision Foundation Model
AI总结:
Morphe基于视觉基础模型,通过联合训练和智能分组丢弃技术,实现高保真度、低带宽的实时视频流媒体传输。
摘要:
视频流媒体是互联网的基础服务,但其质量在带宽受限和偏远地区等网络条件较差时仍无法得到保障。现有研究主要集中在两个方向:传统像素编码流媒体接近极限,难以进一步压缩;新兴的神经增强或生成式流媒体在延迟和视觉保真度上存在不足,阻碍了其实际部署。受最近视觉基础模型(VFM)成功启发,我们致力于利用VFM强大的视频理解和处理能力,实现通用性、高保真度和抗损失能力,以实现更高压缩率的实时视频流媒体。我们提出了首个革命性的范式,使基于VFM的端到端生成视频流媒体朝着这一目标迈进。具体而言,Morphe采用视觉标记符和可变分辨率时空优化的联合训练,在模拟网络约束下进行训练。此外,构建了一个稳健的流媒体系统,利用智能分组丢弃技术来抵抗现实网络扰动。大量评估表明,Morphe在带宽节省62.5%的情况下,与H.265相比实现了可比的视觉质量,并在具有挑战性的网络环境中实现了实时、抗损失的视频交付,代表了VFM赋能多媒体流媒体解决方案的里程碑。
英文摘要:
Video streaming is a fundamental Internet service, while the quality still cannot be guaranteed especially in poor network conditions such as bandwidth-constrained and remote areas. Existing works mainly work towards two directions: traditional pixel-codec streaming nearly approaches its limit and is hard to step further in compression; the emerging neural-enhanced or generative streaming usually fall short in latency and visual fidelity, hindering their practical deployment. Inspired by the recent success of vision foundation model (VFM), we strive to harness the powerful video understanding and processing capacities of VFM to achieve generalization, high fidelity and loss resilience for real-time video streaming with even higher compression rate. We present the first revolutionized paradigm that enables VFM-based end-to-end generative video streaming towards this goal. Specifically, Morphe employs joint training of visual tokenizers and variable-resolution spatiotemporal optimization under simulated network constraints. Additionally, a robust streaming system is constructed that leverages intelligent packet dropping to resist real-world network perturbations. Extensive evaluation demonstrates that Morphe achieves comparable visual quality while saving 62.5\% bandwidth compared to H.265, and accomplishes real-time, loss-resilient video delivery in challenging network environments, representing a milestone in VFM-enabled multimedia streaming solutions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。