LongLive：实时交互式长视频生成
LongLive: Real-time Interactive Long Video Generation
AI总结：
LongLive提出帧级自回归框架，通过KV重缓存、流式长调优和帧汇聚，实现实时交互式长视频生成，支持单卡240秒视频。
摘要：
我们提出了LongLive，一个用于实时和交互式长视频生成的帧级自回归（AR）框架。长视频生成在效率和质量方面都面临挑战。扩散和扩散-强制模型可以生成高质量视频，但由于双向注意力而效率低下。因果注意力AR模型支持KV缓存以加快推理，但在长视频训练期间由于内存挑战，在长视频上往往质量下降。此外，除了基于静态提示的生成之外，交互能力（如流式提示输入）对于动态内容创建至关重要，使用户能够实时引导叙事。这种交互需求显著增加了复杂性，尤其是在确保提示切换期间的视觉一致性和语义连贯性方面。为了解决这些挑战，LongLive采用因果、帧级AR设计，集成了KV重新缓存机制，用新提示刷新缓存状态以实现平滑、连贯的切换；流式长调优以支持长视频训练并对齐训练和推理（训练长-测试长）；以及短窗口注意力配合帧级注意力汇聚（简称帧汇聚），在实现更快生成的同时保持长程一致性。凭借这些关键设计，LongLive仅用32个GPU天就将一个1.3B参数的短视频模型微调为分钟级生成。在推理时，LongLive在单个NVIDIA H100上维持20.7 FPS，在VBench上对短视频和长视频均取得强劲性能。LongLive支持在单个H100 GPU上生成长达240秒的视频。LongLive还支持INT8量化推理，仅产生轻微质量损失。
英文摘要：
We present LongLive, a frame-level autoregressive (AR) framework for real-time and interactive long video generation. Long video generation presents challenges in both efficiency and quality. Diffusion and Diffusion-Forcing models can produce high-quality videos but suffer from low efficiency due to bidirectional attention. Causal attention AR models support KV caching for faster inference, but often degrade in quality on long videos due to memory challenges during long-video training. In addition, beyond static prompt-based generation, interactive capabilities, such as streaming prompt inputs, are critical for dynamic content creation, enabling users to guide narratives in real time. This interactive requirement significantly increases complexity, especially in ensuring visual consistency and semantic coherence during prompt transitions. To address these challenges, LongLive adopts a causal, frame-level AR design that integrates a KV-recache mechanism that refreshes cached states with new prompts for smooth, adherent switches; streaming long tuning to enable long video training and to align training and inference (train-long-test-long); and short window attention paired with a frame-level attention sink, shorten as frame sink, preserving long-range consistency while enabling faster generation. With these key designs, LongLive fine-tunes a 1.3B-parameter short-clip model to minute-long generation in just 32 GPU-days. At inference, LongLive sustains 20.7 FPS on a single NVIDIA H100, achieves strong performance on VBench in both short and long videos. LongLive supports up to 240-second videos on a single H100 GPU. LongLive further supports INT8-quantized inference with only marginal quality loss.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
