Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the performance of these models heavily relies on design choices such as network structures, training data, and training strategies, and these choices have not been extensively discussed in the literature, making it difficult to quantify progress in this field. To address this issue, this paper presents a systematic and comprehensive study, quantitatively and qualitatively, on training such models. We implement over 20 variants with controlled settings. Concretely, for network structures, we compare different LLM backbones and model designs. For training data, we investigate the impact of data and sampling strategies. For instructions, we explore the influence of diversified prompts on the instruction-following ability of the trained models. For benchmarks, we contribute the first, to our best knowledge, comprehensive evaluation set including both image and video tasks through crowd-sourcing. Based on our findings, we present Lynx, which performs the most accurate multi-modal understanding while keeping the best multi-modal generation ability compared to existing open-sourced GPT4-style models.
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）；Kuaishou Technology（快手科技）
Multimodal and multi-domain stylization are two important problems in the field of image style transfer. Currently, there are few methods that can perform both multimodal and multi-domain stylization simultaneously. In this paper, we propose a unified framework for multimodal and multi-domain style transfer with the support of both exemplar-based reference and randomly sampled guidance. The key component of our method is a novel style distribution alignment module that eliminates the explicit distribution gaps between various style domains and reduces the risk of mode collapse. The multimodal …
Northeastern University（东北大学）；Adobe Research（奥多比研究院）；ByteDance AI Lab（字节跳动人工智能实验室）
An assumption widely used in recent neural style transfer methods is that image styles can be described by global statics of deep features like Gram or covariance matrices. Alternative approaches have represented styles by decomposing them into local pixel or neural patches. Despite the recent progress, most existing methods treat the semantic patterns of style image uniformly, resulting unpleasing results on complex styles. In this paper, we introduce a more flexible and general universal style transfer technique: multimodal style transfer (MST). MST explicitly considers the matching of seman…
Generating Multimodal Images with GAN: Integrating Text, Image, and Style
Northeastern University（东北大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Tsinghua University（清华大学）；Montclair State University（蒙特克莱尔州立大学）；Northern Arizona University（北亚利桑那大学）
提出一种基于 GAN 的多模态图像生成方法，通过文本、图像与风格编码模块及多重一致性损失，在多个公开数据集上显著提升生成图像的清晰度与风格一致性。
Unpaired multimodal image-to-image translation is a task of translating a given image in a source domain into diverse images in the target domain, overcoming the limitation of one-to-one mapping. Existing multimodal translation models are mainly based on the disentangled representations with an image reconstruction loss. We propose two approaches to improve multimodal translation quality. First, we use a content representation from the source domain conditioned on a style representation from the target domain. Second, rather than using a typical image reconstruction loss, we design MILO (Mutua…
Fashion is a complex social phenomenon. People follow fashion styles from demonstrations by experts or fashion icons. However, for machine agent, learning to imitate fashion experts from demonstrations can be challenging, especially for complex styles in environments with high-dimensional, multimodal observations. Most existing research regarding fashion outfit composition utilizes supervised learning methods to mimic the behaviors of style icons. These methods suffer from distribution shift: because the agent greedily imitates some given outfit demonstrations, it can drift away from one style…
MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
University of Science and Technology Beijing（北京科技大学）；Alibaba DAMO Academy（阿里巴巴达摩院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；University of Sydney（悉尼大学）
UC Santa Barbara（加州大学圣巴巴拉分校）；UC Santa Cruz（加州大学圣克鲁兹分校）；UC San Diego（加州大学圣地亚哥分校）；Google（谷歌）
One of the most challenging topics in Natural Language Processing (NLP) is visually-grounded language understanding and reasoning. Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment. Due to the lack of human-annotated instructions that illustrate intricate urban scenes, outdoor VLN remains a challenging task to solve. This paper introduces a Multimodal Text Style Transfer (MTST) learning approach and leverages external multimodal resources to mitigate data scarcity in outdoor navigation ta…
Hanoi University of Science and Technology（河内理工大学）；University of Aizu（会津大学）；Eastern International University（东方国际大学）
Recently, style transfer is a research area that attracts a lot of attention, which transfers the style of an image onto a content target. Extensive research on style transfer has aimed at speeding up processing or generating high-quality stylized images. Most approaches only produce an output from a content and style image pair, while a few others use complex architectures and can only produce a certain number of outputs. In this paper, we propose a simple method for representing style features in many ways called Deep Feature Rotation (DFR), while not only producing diverse outputs but also …
BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；Northeastern University（东北大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Zhejiang University（浙江大学）
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；Adobe Research（Adobe 研究院）
Transferring artistic styles onto everyday photographs has become an extremely popular task in both academia and industry. Recently, offline training has replaced on-line iterative optimization, enabling nearly real-time stylization. When those stylization networks are applied directly to high-resolution images, however, the style of localized regions often appears less similar to the desired artistic style. This is because the transfer process fails to capture small, intricate textures and maintain correct texture scales of the artworks. Here we propose a multimodal convolutional neural netwo…
FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow
School of Computer Science, Peking University（北京大学计算机科学系）；Technical University of Munich（慕尼黑技术大学）；Beijing Jiaotong University（北京交通大学）；Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究院）；Theta Labs, Inc.（Theta Labs公司）；Beijing Normal University（北京师范大学）
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what) and mouse traces over an empty canvas (the where) to express the…
We introduce a multimodal visual-textual search refinement method for fashion garments. Existing search engines do not enable intuitive, interactive, refinement of retrieved results based on the properties of a particular product. We propose a method to retrieve similar items, based on a query item image and textual refinement properties. We believe this method can be leveraged to solve many real-life customer scenarios, in which a similar item in a different color, pattern, length or style is desired. We employ a joint embedding training scheme in which product images and their catalog textua…
Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervi…
Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation
Wangxuan Institute of Computer Technology, Peking University（北京大学王学轩计算机技术研究所）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the machine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important compon…
Korea University（高丽大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Columbia University（哥伦比亚大学）；Microsoft Research Asia（微软亚洲研究院）
Recently, image-to-image translation has obtained significant attention. Among many, those approaches based on an exemplar image that contains the target style information has been actively studied, due to its capability to handle multimodality as well as its applicability in practical use. However, two intrinsic problems exist in the existing methods: what and where to transfer. First, those methods extract style from an entire exemplar which includes noisy information, which impedes a translation model from properly extracting the intended style of the exemplar. That is, we need to carefully…
Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose UVIT, a novel unsupervised video-to-video translation model. Our model decomposes the style and the content, uses the specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units. The style-content decomposition mechanism enables us to achieve style consistent video translation results as well as provides us wit…
E-commerce platforms surface interesting products largely through product recommendations that capture users' styles and aesthetic preferences. Curating recommendations as a complete complementary set, or assortment, is critical for a successful e-commerce experience, especially for product categories such as furniture, where items are selected together with the overall theme, style or ambiance of a space in mind. In this paper, we propose two visually-aware recommender systems that can automatically curate an assortment of living room furniture around a couple of pre-selected seed pieces for …
Cornell University（康奈尔大学）；Delft University of Technology（代尔夫特理工大学）
A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flipping or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. However, a stylized photograph is not quite the same as an artist-created painting. Artists depict perceptually meaningful cues in paintings so that humans can recognize salient components in scenes, an …
WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation
King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹大学科学与技术大学）；Lanzhou University（兰州大学）；Meta AI；The University of Sydney（悉尼大学）；IHPC, A*STAR（IHPC，A*STAR）
UC Santa Barbara（加州大学圣巴巴拉分校）；Meta；NEC Laboratories America（NEC美国实验室）
Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which re…
Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model se…
CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Pengcheng Laboratory（鹏城实验室）；MAIS, Institute of Automation, Chinese Academy of Sciences（自动化所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；ByteDance Inc.（字节跳动公司）；National Cheng Kung University（国立成功大学）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
It seems easy to imagine a photograph of the Eiffel Tower painted in the style of Vincent van Gogh's 'The Starry Night', but upon introspection it is difficult to precisely define what this would entail. What visual elements must an image contain to represent the 'content' of the Eiffel Tower? What visual elements of 'The Starry Night' are caused by van Gogh's 'style' rather than his decision to depict a village under the night sky? Precisely defining 'content' and 'style' is a central challenge of designing algorithms for artistic style transfer, algorithms which can recreate photographs usin…
