We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs. Extensive experiments demonstrate that this recipe is highly effective for multi-modal models. CM3Leon achieves state-of-the-art performance in text-to-image generation with 5x less training compute than comparable methods (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate unprecedented levels of controllability in tasks ranging from language-guided image editing to image-controlled generation and segmentation.
Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to pre…
Physical Autoregressive Model for Robotic Manipulation without Action Pretraining
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；X-Era AI Lab（X-Era人工智能实验室）；Guangdong University of Technology（广东工业大学）
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Department of Linguistics and Translation, City University of Hong Kong（香港城市大学语言学及翻译学系）
Microsoft Research AI（微软研究院AI）；Peking University（北京大学）；University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments. In this paper, we report two simple but highly effective methods to address these challenges and lead to a new state-of-the-art performance. First, we adapt large-scale pretrained language models to learn text representations that generalize better to previously unseen instructions. Second, we propose a stochastic sampling scheme to reduce the considerable gap between the expert actio…
Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic content. To address this challenge, we propose two novel pretraining data augmentation strategies that significantly reduce model toxicity without compromising its utility. Our two strategies are: (1) MEDA: adds raw toxicity score as meta-data to the pretraining samples, and (2) INST: adds instructions to those samples indicating their toxicity. Our results indicate that our b…
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
Gaoling School of AI, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部）；Ant Group（蚂蚁集团）
Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying o…
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院）；Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系）；Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院）；School of Humanities, Tallinn University（塔林大学人文学院）
Carnegie Mellon University（卡内基梅隆大学）；Google AI Brain Team（谷歌人工智能大脑团队）
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order an…
Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement
Guru Gobind Singh Indraprastha University, India（古鲁·戈宾德·辛格·印度普拉斯塔大学）；Artificial Intelligence Institute, University of South Carolina, USA（人工智能研究所，南卡罗来纳大学）；Indian Institute of Technology, Kanpur, India（印度理工学院，坎浦尔）；Indian AI Research Organization, India（印度人工智能研究组织）
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
Shanghai Key Lab of Intelligent Information Processing, Fudan University（复旦大学上海智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Youtu Lab, Tencent（腾讯优图实验室）；Meta AI；Shanghai AI Laboratory（上海人工智能实验室）
Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models. However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised m…
