在大量数据语料库上训练的具有数十亿参数的基础模型在多个领域展示了非凡的能力。然而，由于其单体结构，增强它们或赋予新技能具有挑战性且成本高昂。另一方面，由于它们的适应能力，这些模型的新实例正被训练以适应新的领域和任务。在本研究中，我们探讨了将现有基础模型与更具体的模型进行高效且实用的组合以实现新能力的问题。为此，我们提出了CALM（Composition to Augment Language Models，组合增强语言模型），该方法在模型之间引入交叉注意力来组合它们的表示并实现新能力。CALM的主要特点是：（i）通过“重用”现有LLM以及少量额外参数和数据，在新任务上扩展LLM；（ii）保持现有模型权重不变，从而保留现有能力；（iii）适用于多样化的领域和设置。我们展示了将PaLM2-S与在低资源语言上训练的较小模型相结合，在翻译成英语和低资源语言算术推理等任务上实现了13%的绝对提升。类似地，当PaLM2-S与特定代码模型结合时，在代码生成和解释任务上比基础模型实现了40%的相对提升——与完全微调的对应模型相当。
英文摘要：
Foundational models with billions of parameters which have been trained on large corpora of data have demonstrated non-trivial skills in a variety of domains. However, due to their monolithic structure, it is challenging and expensive to augment them or impart new skills. On the other hand, due to their adaptation abilities, several new instances of these models are being trained towards new domains and tasks. In this work, we study the problem of efficient and practical composition of existing foundation models with more specific models to enable newer capabilities. To this end, we propose CALM -- Composition to Augment Language Models -- which introduces cross-attention between models to compose their representations and enable new capabilities. Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using' existing LLMs along with a few additional parameters and data, (ii) Existing model weights are kept intact, and hence preserves existing capabilities, and (iii) Applies to diverse domains and settings. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13\% on tasks like translation into English and arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40\% over the base model for code generation and explanation tasks -- on-par with fully fine-tuned counterparts.
NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）
Multi-Sourced Compositional Generalization in Visual Question Answering
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学，中国）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, China（广东机器感知与智能计算实验室，深圳MSU-BIT大学，中国）
Physics Augmented Machine Learning Discovery of Composition-Dependent Constitutive Laws for 3D Printed Digital Materials
Sibley School of Mechanical and Aerospace Engineering, Cornell University（科林斯大学机械与航空航天工程学院）；Department of Materials Science and Engineering, Technion - Israel Institute of Technology（技术学院-以色列理工学院材料科学与工程系）；Sandia National Laboratories（桑地亚国家实验室）；Center for Applied Mathematics, Ithaca, NY 14850（应用数学中心）；Pasteur Labs, Brooklyn, NY 11205（帕斯特实验室）
