arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21669cs.CV
收藏

研究视频扩散模型中的记忆现象

Investigating Memorization in Video Diffusion Models

School of Computer Science, Faculty of Engineering, The University of Sydney, Australia(悉尼大学计算机科学学院,工程学院);School of Physics, Mathematics and Computing, The University of Western Australia, Australia(西澳大学物理、数学与计算学院);Center for Research in Computer Vision, University of Central Florida, USA(佛罗里达中央大学计算机视觉研究中心)

Chen Chen, Enhuai Liu, Daochang Liu, Mubarak Shah, Chang Xu

AI总结:

本文首次定义并评估了视频扩散模型(VDMs)中的内容与运动记忆现象,通过构建易触发记忆的文本提示数据集和新评估指标,系统揭示了记忆现象在各类VDMs中的普遍性,并提出了高效的隐私检测策略。

摘要:

扩散模型广泛应用于图像和视频生成,但面临一个显著局限:在推理阶段存在记忆并复现训练数据的风险,可能生成未经授权的版权内容。以往研究主要关注图像扩散模型(IDMs),而视频扩散模型(VDMs)仍鲜有探讨。为填补这一空白,我们首先以注重隐私保护且适用于所有生成类型的实用方式,正式定义了VDMs中的两类记忆现象(内容记忆和运动记忆)。随后,我们引入了专门设计用于分别评估VDMs中内容记忆和运动记忆的新指标。此外,我们精心构建了一个文本提示数据集,这些提示在作为VDMs的条件时最容易触发记忆现象。利用这些提示,我们从各种开源VDMs中生成了多样化视频,并成功从每个测试模型中提取出大量训练视频。通过应用我们提出的指标,我们系统分析了各种预训练VDMs(包括文本条件和无条件模型)在多种数据集上的记忆现象。我们的综合研究表明,记忆现象在所有测试的VDMs中普遍存在,这表明VDMs除了记忆视频数据集外,也能记忆图像训练数据。最后,我们针对内容记忆和运动记忆提出了高效且有效的检测策略,为改善VDMs的隐私提供了基础方法。

英文摘要:

Diffusion models, widely used for image and video generation, face a significant limitation: the risk of memorizing and reproducing training data during inference, potentially generating unauthorized copyrighted content. While prior research has focused on image diffusion models (IDMs), video diffusion models (VDMs) remain underexplored. To address this gap, we first formally define the two types of memorization in VDMs (content memorization and motion memorization) in a practical way that focuses on privacy preservation and applies to all generation types. We then introduce new metrics specifically designed to separately assess content and motion memorization in VDMs. Additionally, we curate a dataset of text prompts that are most prone to triggering memorization when used as conditioning in VDMs. By leveraging these prompts, we generate diverse videos from various open-source VDMs, successfully extracting numerous training videos from each tested model. Through the application of our proposed metrics, we systematically analyze memorization across various pretrained VDMs, including text-conditional and unconditional models, on a variety of datasets. Our comprehensive study reveals that memorization is widespread across all tested VDMs, indicating that VDMs can also memorize image training data in addition to video datasets. Finally, we propose efficient and effective detection strategies for both content and motion memorization, offering a foundational approach for improving privacy in VDMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑