鲁棒的声音引导图像编辑
Robust Sound-Guided Image Manipulation
AI总结：
针对文本提示语义表达受限的问题，本文提出将声音融入图文联合嵌入空间，并通过直接隐式优化实现声音引导的图像编辑，实验证明其效果优于现有文本与声音引导方法。
摘要：
近期的成功表明，图像可以通过文本提示进行编辑，例如，在文本输入“下雨”的驱动下，晴天的风景场景被编辑成同一天气场景的雨天版本。这些方法通常采用基于 StyleCLIP 的图像生成器，该生成器利用多模态（文本和图像）嵌入空间。然而，我们观察到，此类文本输入在提供和合丰富语义线索方面常常存在瓶颈，例如区分大雨与伴有雷暴的雨。为了解决这一问题，我们主张利用另一种模态——声音，它在图像编辑中具有显著优势，因为与文本相比，声音能够传达更加多样的语义线索（生动的情感或自然界的动态表达）。本文提出一种新颖方法，首先用声音扩展图文联合嵌入空间，然后采用直接隐式优化方法，根据音频输入（例如雨声）对给定图像进行编辑。大量实验表明，与最先进的文本引导和声音引导图像编辑方法相比，我们的声音引导图像编辑方法在语义和视觉上均能产生更合理的编辑结果，人工评估也进一步证实了这一点。下游任务评估还表明，我们学习到的图像-文本-声音联合嵌入空间能够有效编码声音输入。
英文摘要：
Recent successes suggest that an image can be manipulated by a text prompt, e.g., a landscape scene on a sunny day is manipulated into the same scene on a rainy day driven by a text input "raining". These approaches often utilize a StyleCLIP-based image generator, which leverages multi-modal (text and image) embedding space. However, we observe that such text inputs are often bottlenecked in providing and synthesizing rich semantic cues, e.g., differentiating heavy rain from rain with thunderstorms. To address this issue, we advocate leveraging an additional modality, sound, which has notable advantages in image manipulation as it can convey more diverse semantic cues (vivid emotions or dynamic expressions of the natural world) than texts. In this paper, we propose a novel approach that first extends the image-text joint embedding space with sound and applies a direct latent optimization method to manipulate a given image based on audio input, e.g., the sound of rain. Our extensive experiments show that our sound-guided image manipulation approach produces semantically and visually more plausible manipulation results than the state-of-the-art text and sound-guided image manipulation methods, which are further confirmed by our human evaluations. Our downstream task evaluations also show that our learned image-text-sound joint embedding space effectively encodes sound inputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
