arXiv 2604.080842026-04-09cs.CV

DiffVC: 一种基于扩散模型的非自回归框架用于视频描述生成

DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning

School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Beijing University Of Technology（北京工业大学）

Junbo Wang, Liangyu Fu, Yuke Li, Yining Zhu, Ya Jing, Xuecheng Wu, Jiangbin Zheng

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DiffVC，一种基于扩散模型的非自回归框架，解决视频描述生成中自回归方法速度慢和累积误差问题，通过并行解码和判别性条件扩散模型提升生成质量，实验表明其在多个数据集上优于非自回归方法并接近自回归方法。

摘要：

当前视频描述生成方法通常使用编码器-解码器结构进行自回归生成。然而，自回归方法存在固有局限，如生成速度慢和累积误差大。此外，非自回归方法由于缺乏足够的多模态交互建模而存在生成质量缺陷。因此，我们提出了一种基于扩散模型的视频描述生成非自回归框架（DiffVC）以解决这些问题。其并行解码可有效解决生成速度和累积误差问题。同时，我们提出的判别性条件扩散模型可生成更高质量的文本描述。具体而言，我们首先将视频编码为视觉表示。在训练过程中，向真实描述的文本表示添加高斯噪声。然后，通过判别性去噪器生成新的文本表示，以视觉表示作为条件约束。最后，将新的文本表示输入非自回归语言模型以生成描述。在推理过程中，我们直接从高斯分布采样噪声进行生成。在MSVD、MSR-VTT和VATEX数据集上的实验表明，我们的方法在多个数据集上优于之前的非自回归方法，并且在自回归方法上达到可比性能，例如在CIDEr上最大提升9.9，在B@4上提升2.6，同时生成速度更快。源代码将很快提供。

英文摘要：

Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on Diffusion model for Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。