The field of multimodal research focusing on the comprehension and creation of both images and text has witnessed significant strides. This progress is exemplified by the emergence of sophisticated models dedicated to image captioning at scale, such as the notable Flamingo model and text-to-image generative models, with DALL-E serving as a prominent example. An interesting question worth exploring in this domain is whether Flamingo and DALL-E understand each other. To study this question, we propose a reconstruction task where Flamingo generates a description for a given image and DALL-E uses this description as input to synthesize a new image. We argue that these models understand each other if the generated image is similar to the given image. Specifically, we study the relationship between the quality of the image reconstruction and that of the text generation. We find that an optimal description of an image is one that gives rise to a generated image similar to the original one. The finding motivates us to propose a unified framework to finetune the text-to-image and image-to-text models. Concretely, the reconstruction part forms a regularization loss to guide the tuning of the models. Extensive experiments on multiple datasets with different image captioning and image generation models validate our findings and demonstrate the effectiveness of our proposed unified framework. As DALL-E and Flamingo are not publicly available, we use Stable Diffusion and BLIP in the remaining work. Project website: https://dalleflamingo.github.io.
OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understanding
Xiamen University(厦门大学);University of Electronic Science and Technology of China(电子科技大学);Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院,清华大学);Xidian University(西安电子科技大学)
Can Artificial Intelligence Reconstruct Ancient Mosaics?
Universidad Antonio de Nebrija(安东尼奥·德·内夫里哈大学);Universidad de Valladolid(巴利亚多利德大学);Universidad Politécnica de Madrid(马德里理工大学);Northeastern University(东北大学)
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
University of Twente(特文特大学);University of Technology Nuremberg (UTN)(纽伦堡工业大学(UTN));University of Sheffield(谢菲尔德大学);Harbin Institute of Technology(哈尔滨工业大学);University of Mannheim(曼海姆大学);Technische Universität Dresden(德累斯顿工业大学)
From Noise to Nuance: Advances in Deep Generative Image Models
Georgia Institute of Technology(佐治亚理工学院);AppCubic(AppCubic公司);JTB Technology Corp.(JTB科技公司);Indiana University(印第安纳大学);Purdue University(普渡大学);The University of Texas at Dallas(德州大学达拉斯分校);University of Liverpool(利物浦大学);Emory University(埃默里大学);National Taiwan Normal University(台湾师范大学)
LAION-5B: An open large-scale dataset for training next generation image-text models
LAION;UC Berkeley(加州大学伯克利分校);Gentec Data;TU Darmstadt(达姆施塔特工业大学);University of Washington(华盛顿大学);Technical University of Munich(慕尼黑工业大学);Stability AI;EleutherAI;Juelich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Juelich (FZJ)(于利希研究中心)
PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
OpenGVLab(开放智能机器研究实验室);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);University of Michigan(密歇根大学);Tongji University(同济大学);Nanjing University(南京大学);The University of Hong Kong(香港大学)
Multiview Compressive Coding for 3D Reconstruction
FAIR, Meta AI(元人工智能公司FAIR实验室)
视觉识别的核心目标是从单个图像中理解对象和场景。由于大规模学习和通用表示,2D识别已经取得了巨大的进展。相比之下,3D提出了新的挑战,源于图像中未描绘的遮挡。先前的工作试图通过从多个视图推断或依赖于稀缺的CAD模型和类别特定的先验来克服这些问题,这些先验阻碍了扩展到新的设置。在这项工作中,我们探索单视图三维重建学习的启发,在自监督学习的进步概括表示。我们介绍了一个简单的框架,该框架在单个对象或整个场景的3D点上操作,再加上来自不同RGB-D视频的类别不可知的大规模训练。我们的模型,多视图压缩编码(MCC),学习压缩输入的外观和几何形状,通过查询3D感知解码器来预测3D结构。MCC的通用性和效率使其能够从大规模和多样化的数据源中学习,并对DALL $\cdot$ E 2想象的新对象或用iPhone在野外捕获的新对象具有很强的泛化能力。