StreamChar: 基于解耦编排的长时程流式角色音频-视频生成
StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
AI总结：
提出StreamChar流式框架，通过LLM编排器与联合音频-视频DiT解耦长时程编排与短窗去噪，实现实时、稳定、高质量的角色动画生成。
摘要：
实时流式联合音频-视频生成用于角色动画需要生成器说出请求的文本、跨块保持视觉身份并在严格的播放预算内运行。这些要求难以同时满足：逐块自回归生成会累积文本-音频错位和视觉漂移，而低延迟所需的少步蒸馏通常会降低空间多样性和时间质量。我们提出StreamChar，一种将长时程编排与短窗音频-视频去噪分离的流式框架。基于LLM的编排器使用文本和历史上下文生成帧对齐的音频条件，联合音频-视频DiT在参考和运动帧条件下执行局部双向去噪。为高效部署，我们使用两阶段蒸馏流程，首先压缩采样器，然后在在线块展开下微调学生模型。进度感知指针在展开训练期间将部分文本与生成的音频对齐，而汇块记忆提供持久视觉锚点以减少长时程漂移。在短片段和长时程协议上的实验表明，StreamChar在单个H100 GPU上实时运行，与最近的联合和音频驱动基线相比，在文本保真度、音视频同步、视觉质量和流式稳定性方面提供了有利的系统级权衡。
英文摘要：
Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
