Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized representations for videos and apply a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. To improve video quality and consistency, we propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. We introduce text augmentation to improve the robustness of the textual representation and diversity of generated videos. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images. It can generate much longer sequences than the one used for training. In addition, our model can extract visual information as suggested by the text prompt, e.g., "an object in image one is moving northeast", and generate corresponding videos. We run evaluations on three public datasets and a newly collected dataset labeled with facial attributes, achieving state-of-the-art generation results on all four.
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
We are creating multimedia contents everyday and everywhere. While automatic content generation has played a fundamental challenge to multimedia community for decades, recent advances of deep learning have made this problem feasible. For example, the Generative Adversarial Networks (GANs) is a rewarding approach to synthesize images. Nevertheless, it is not trivial when capitalizing on GANs to generate videos. The difficulty originates from the intrinsic structure where a video is a sequence of visually coherent and semantically dependent frames. This motivates us to explore semantic and tempo…
UC Santa Barbara（加州大学圣巴巴拉分校）；Meta；NEC Laboratories America（NEC美国实验室）
Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which re…
Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Synthesis with Modality and Metadata Conditioning
Department of Psychology & Neuroscience, Auckland University of Technology（心理学与神经科学系，奥克兰技术大学）；Department of Psychiatry, University of Melbourne（精神病学系，墨尔本大学）
Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matching
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Google Research（谷歌研究）；Sony Research, India（索尼印度研究）；Indian Institute of Technology, Roorkee（印度理工学院罗尔基分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
University of Guelph（圭尔夫大学）；Microsoft Research（微软研究院）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；University of Montreal（蒙特利尔大学）；Canadian Institute for Advanced Research（加拿大高级研究院）
Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from available conditioning information in one step. One practical extension beyond one-step generation is a system that generates an image iteratively, conditioned on ongoing linguistic input or feedback. This is significantly more challenging than one-step generation tasks, as such a system must understand the contents of its generated images with respect to the feedback history, the current feedback, as well as the interact…
Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment
University of Bari Aldo Moro（巴里大学阿尔多·莫罗大学）；Catalonia's Telecommunications Technology Centre（加泰罗尼亚电信技术中心）；University of Picardie Jules Verne（皮卡第大学朱利·瓦内大学）
Several deep learning methods have been proposed for completing partial data from shape acquisition setups, i.e., filling the regions that were missing in the shape. These methods, however, only complete the partial shape with a single output, ignoring the ambiguity when reasoning the missing geometry. Hence, we pose a multi-modal shape completion problem, in which we seek to complete the partial shape with multiple outputs by learning a one-to-many mapping. We develop the first multimodal shape completion method that completes the partial shape via conditional generative modeling, without req…
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
GeoBridge: Decoupled Semantic Conditioning for Generative Image Geolocalization
University of Chinese Academy of Sciences（中国科学院大学）；School of Advanced Interdisciplinary Sciences（先进交叉科学学院）；School of Electronic, Electrical and Communication Engineering（电子电气与通信工程学院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）
Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School（马萨诸塞州总医院与哈佛医学院戈登医学影像中心）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
Allen Institute for AI（艾伦人工智能研究所）；Cornell University（康奈尔大学）
Modeling expressive cross-modal interactions seems crucial in multimodal tasks, such as visual question answering. However, sometimes high-performing black-box algorithms turn out to be mostly exploiting unimodal signals in the data. We propose a new diagnostic tool, empirical multimodally-additive function projection (EMAP), for isolating whether or not cross-modal interactions improve performance for a given model on a given task. This function projection modifies model predictions so that cross-modal interactions are eliminated, isolating the additive, unimodal structure. For seven image+te…
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what) and mouse traces over an empty canvas (the where) to express the…
Non-Fungible Tokens (NFTs) represent deeds of ownership, based on blockchain technologies and smart contracts, of unique crypto assets on digital art forms (e.g., artworks or collectibles). In the spotlight after skyrocketing in 2021, NFTs have attracted the attention of crypto enthusiasts and investors intent on placing promising investments in this profitable market. However, the NFT financial performance prediction has not been widely explored to date. In this work, we address the above problem based on the hypothesis that NFT images and their textual descriptions are essential proxies to p…
Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
School of Computer Science, Beijing Institute of Technology（北京理工大学计算机科学学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Xiaohongshu Inc（小红书公司）
Stitch and Tell通过结构化空间监督提升视觉-语言模型的空间理解能力，有效缓解空间幻觉并提高相关任务性能。
