Investigating Memorization in Video Diffusion Models
School of Computer Science, Faculty of Engineering, The University of Sydney, Australia(悉尼大学计算机科学学院,工程学院);School of Physics, Mathematics and Computing, The University of Western Australia, Australia(西澳大学物理、数学与计算学院);Center for Research in Computer Vision, University of Central Florida, USA(佛罗里达中央大学计算机视觉研究中心)
Diffusion models, widely used for image and video generation, face a significant limitation: the risk of memorizing and reproducing training data during inference, potentially generating unauthorized copyrighted content. While prior research has focused on image diffusion models (IDMs), video diffusion models (VDMs) remain underexplored. To address this gap, we first formally define the two types of memorization in VDMs (content memorization and motion memorization) in a practical way that focuses on privacy preservation and applies to all generation types. We then introduce new metrics specifically designed to separately assess content and motion memorization in VDMs. Additionally, we curate a dataset of text prompts that are most prone to triggering memorization when used as conditioning in VDMs. By leveraging these prompts, we generate diverse videos from various open-source VDMs, successfully extracting numerous training videos from each tested model. Through the application of our proposed metrics, we systematically analyze memorization across various pretrained VDMs, including text-conditional and unconditional models, on a variety of datasets. Our comprehensive study reveals that memorization is widespread across all tested VDMs, indicating that VDMs can also memorize image training data in addition to video datasets. Finally, we propose efficient and effective detection strategies for both content and motion memorization, offering a foundational approach for improving privacy in VDMs.
Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis
Heidelberg University Hospital(海德堡大学医院);German Centre for Cardiovascular Research(德国心血管研究中心);Heidelberg University(海德堡大学);University Medical Center Mannheim(曼海姆大学医学中心)
TTOM: Test-Time Optimization and Memorization for Compositional Video Generation
National University of Singapore(国立新加坡大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
VidLeaks: Membership Inference Attacks Against Text-to-Video Models
School of Cyber Science and Technology, Shandong University(山东大学网络科学与技术学院);Eyeline Labs(Eyeline实验室);State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)
Generating Novel Scene Compositions from Single Images and Videos
Bosch Center for Artificial Intelligence(博世人工智能中心);University of Tübingen(图宾根大学);University of Bonn(波恩大学);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所)
Generative Ghost: Investigating Ranking Bias Hidden in AI-Generated Videos
State Key Lab of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)
Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data
Leibniz Information Centre for Science and Technology (TIB)(莱布尼茨科学技术信息中心(TIB));L3S Research Center(L3S研究中心);University of Marburg(马尔堡大学);German Broadcasting Archive (DRA)(德国广播档案馆(DRA))
Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
University of Washington(华盛顿大学);Center for Neurotechnology(神经技术中心);University of Washington Institute for Neuroengineering(华盛顿大学神经工程研究所);eScience Institute, University of Washington(华盛顿大学电子科学研究所)
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
The University of Texas at Austin(德克萨斯大学奥斯汀分校)
最近的视觉语言学预训练模型在各种最终任务上显示出有希望的进展,例如图像检索和视频描述。然而,他们在最近提出的Winoground数据集上失败得很惨,该数据集挑战模型匹配配对图像和英文字幕,其中项目被构造为在词汇上重叠但含义不同(例如,“There is a mug in some grass”与“There is some grass in a mug”的对比)。通过使用新的细粒度标签注释数据集,我们表明解决Winoground任务不仅需要组合语言理解,而且还需要许多其他能力,如常识推理或在低分辨率图像中定位小的,焦点外的对象。在本文中,我们通过一系列相关任务(探测任务,图像检索任务),数据增强和手动检查数据集的实验来确定数据集的主要挑战。我们的分析表明,视觉语言学模型的一个主要挑战可能在于融合视觉和文本表征,而不是成分语言理解。我们在https://github.com/ajd12342/why-winoground-hard上发布我们的注释和代码。
DualVision ArthroNav: Investigating Opportunities to Enhance Localization and Reconstruction in Image-based Arthroscopy Navigation via External Cameras
University of Arkansas(阿肯色大学);Arthrex Inc.(阿瑟克斯公司)