基于视觉感知的多模态大语言模型条件编辑扩散的视频率流式风格化：蒸馏UNet + MLLM文本编码器上的非对称批处理推理
Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPU
AI总结：
针对蒸馏扩散模型中文本编码器成为瓶颈的问题，提出一种结合非对称CUDA流水线、编译友好的ControlNet-LLLite重构和周期性条件刷新调度的流式管线，在消费级GPU上实现视频率实时风格化编辑。
摘要：
扩散U-Net的激进蒸馏反转了实时文本到图像流水线的逐帧瓶颈：一旦去噪器成为4步或1步蒸馏的学生模型，文本编码器就成为关键路径。这种反转在视觉感知编辑扩散中最为严重，其中编码器是多模态大语言模型（MLLM）。我们研究了一个0.39B蒸馏编辑U-Net与2.13B MLLM文本编码器（Qwen3-VL）配对的情况，并提出了一种针对该场景的流式管线，该管线围绕三种工程机制构建：非对称侧流/主流CUDA流水线，带有批处理文本编码器摊销（以及可选的静态提示缓存）；一种编译友好的ControlNet-LLLite重构，将整个U-Net +适配器堆栈折叠成单个融合图；以及一个带有钩子子集的周期性条件刷新调度，用于摊销每帧条件成本。在单个消费级RTX 3090 Ti上，512x512分辨率下，管线在批大小B=8时维持27.4 fps，B=16时维持29.6 fps，端到端p50延迟分别约为0.5和1.0秒；相同操作点在RTX 4090上测得54.9 fps，在RTX 5090上测得74.1 fps。我们报告的是视频率流式吞吐量而非交互式低延迟，并将我们的数据与相同堆栈的StreamDiffusion重运行进行对比，作为系统上下文，而非基准优越性声明。对于训练的油画风格，发布的时序适配器在剪辑内噪声中泛化到19个未使用的DAVIS-2017序列和来自七个来源的15个非DAVIS剪辑；对未见风格族的提示级泛化有限，并单独报告。
英文摘要：
Aggressive distillation of the diffusion U-Net inverts the per-frame bottleneck of real-time text-to-image pipelines: once the denoiser is a 4-step or 1-step distilled student, the text encoder becomes the critical path. This inversion is most acute in vision-aware edit diffusion, where the encoder is a multimodal large language model (MLLM). We study a 0.39B distilled edit U-Net paired with a 2.13B MLLM text encoder (Qwen3-VL) and present a streaming pipeline for this regime built on three mechanisms that keep the encoder off the denoiser's critical path rather than shrinking the encoder: asymmetric side-stream / main-stream CUDA pipelining with batched text-encoder amortisation, a compile-friendly ControlNet-LLLite reformulation that folds the whole U-Net + adapter stack into one fused graph, and a periodic conditioning-refresh schedule with a hook subset that amortises the per-frame conditioning cost. On a single consumer RTX 3090 Ti at 512x512 this sustains 27-30 fps over a 480-frame run; at the same operating point steady-state throughput scales to 55 fps on RTX 4090 and 74 fps on RTX 5090. This shows that once distillation is aggressive enough, further gains come from encoder-side systems work rather than further denoiser compression -- the opposite lever from the one the streaming-diffusion literature has optimised to date. We report video-rate streaming throughput, not interactive low latency, and locate our numbers against same-stack StreamDiffusion re-runs as systems context, not a superiority claim. The released oil-painting adapter generalises within in-clip noise to 19 unused DAVIS-2017 sequences and 15 non-DAVIS clips from seven sources; prompt-level generalisation to unseen styles is bounded and reported separately.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
