The correspondence between input text and the generated image exhibits opacity, wherein minor textual modifications can induce substantial deviations in the generated image. While, text embedding, as the pivotal intermediary between text and images, remains relatively underexplored. In this paper, we address this research gap by delving into the text embedding space, unleashing its capacity for controllable image editing and explicable semantic direction attributes within a learning-free framework. Specifically, we identify two critical insights regarding the importance of per-word embedding and their contextual correlations within text embedding, providing instructive principles for learning-free image editing. Additionally, we find that text embedding inherently possesses diverse semantic potentials, and further reveal this property through the lens of singular value decomposition (SVD). These uncovered properties offer practical utility for image editing and semantic discovery. More importantly, we expect the in-depth analyses and findings of the text embedding can enhance the understanding of text-to-image diffusion models.
In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Tencent Youtu Lab(腾讯云图实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Uncovering the human motion pattern: Pattern Memory-based Diffusion Model for Trajectory Prediction
Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia(智能电信软件与多媒体北京市重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学)
Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models
UC, Santa Barbara(加州大学圣巴巴拉分校);Adobe Research(奥多比研究院);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
生成模型在计算机视觉中得到了广泛的研究。近来,扩散模型由于其生成的图像的高质量而引起了相当大的关注。图像生成模型的一个关键期望属性是理清不同属性的能力,这应该能够在不改变语义内容的情况下对风格进行修改,并且修改参数应该推广到不同的图像。以前的研究发现,生成对抗网络(GANs)天生就具有这种解纠缠能力,因此它们可以在不重新训练或微调网络的情况下执行解纠缠图像编辑。在这项工作中,我们探讨是否扩散模型也固有地配备了这样的能力。我们的发现是,对于稳定的扩散模型,通过部分地改变输入文本嵌入的中性描述(例如,“人的照片”)到具有风格的照片(例如,“a photo of person with smile”),同时固定在去噪过程中引入的所有高斯随机噪声,生成的图像可以朝着目标风格修改,而不改变语义内容。基于这一发现,我们进一步提出了一个简单的,轻量级的图像编辑算法,其中两个文本嵌入的混合权重进行了优化的风格匹配和内容保存。整个过程仅涉及优化大约50个参数,并且不微调扩散模型本身。实验表明,该方法可以修改广泛的属性,性能优于基于扩散模型的图像编辑算法,需要微调。优化后的权重可以很好地推广到不同的图像。我们的代码可在https://github.com/UCSB-NLP-Chang/DiffusionDisentanglement上公开获取。
Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Models
University of Georgia(佐治亚大学);Carnegie Mellon University(卡内基梅隆大学);Northeastern University(东北大学);Stevens Institute of Technology(史蒂文斯理工学院);University of Arizona(亚利桑那大学)
Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusion
Department of Mathematics University of California, San Diego(数学系 加州大学圣地亚哥分校);National Security Directorate Pacific Northwest National Laboratory(国家安全局 西部国家实验室);Earth and Biological Sciences Directorate Pacific Northwest National Laboratory(地球与生命科学局 西部国家实验室)
Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);NYU Shanghai(纽约大学上海分校);Alibaba-NTU Singapore Joint Research Institute(阿里-国立新加坡大学联合研究机构);University of the Chinese Academy of Sciences(中国科学院大学);Center for Data Science(数据科学中心);New York University(纽约大学)
Stimulating Diffusion Model for Image Denoising via Adaptive Embedding and Ensembling
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);University of Science and Technology of China(中国科学技术大学);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)