Configurable Foundation Models: Building LLMs from a Modular Perspective
Tsinghua University（清华大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Carnegie Mellon University（卡内基梅隆大学）；ModelBest Inc.（模百科技）；Princeton University（普林斯顿大学）；National University of Singapore（新加坡国立大学）；Stanford University（斯坦福大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Renmin University of China（中国人民大学）
大语言模型（LLMs）的进展近期揭示了与其庞大参数需求相关的计算效率和持续可扩展性挑战，使得这些模型在计算资源有限的设备上以及需要多种能力的场景中的应用和演进日益困难。受人类大脑模块化结构的启发，将LLMs分解为众多功能模块的趋势日益明显，从而允许仅使用部分模块进行推理，并动态组装模块以处理复杂任务，例如专家混合（mixture-of-experts）。为了突出模块化方法固有的效率和可组合性，我们创造了“brick”（砖块）这一术语来表示每个功能模块，并将这种模块化结构命名为可配置基础模型（configurable foundation models）。在本文中，我们对可配置基础模型的构建、利用和局限性进行了全面的概述与调研。我们首先将模块形式化为两类：涌现砖块（emergent bricks）——在预训练阶段自然形成的功能性神经元分区，以及定制砖块（customized bricks）——通过额外的后训练构建的砖块，用于提升LLMs的能力和知识。基于多样化的功能砖块，我们进一步提出了四种面向砖块的操作：检索与路由、合并、更新和增长。这些操作允许根据指令对LLMs进行动态配置，以处理复杂任务。为验证我们的观点，我们对广泛使用的LLMs进行了实证分析。我们发现FFN层遵循模块化模式，表现出神经元的功能特化和功能性神经元分区。最后，我们指出了若干开放问题和未来研究方向。总体而言，本文旨在为现有LLM研究提供一种全新的模块化视角，并启发未来更高效、更具可扩展性的基础模型的创建。
英文摘要：
Advancements in LLMs have recently unveiled challenges tied to computational efficiency and continual scalability due to their requirements of huge parameters, making the applications and evolution of these models on devices with limited computation resources and scenarios requiring various abilities increasingly cumbersome. Inspired by modularity within the human brain, there is a growing tendency to decompose LLMs into numerous functional modules, allowing for inference with part of modules and dynamic assembly of modules to tackle complex tasks, such as mixture-of-experts. To highlight the inherent efficiency and composability of the modular approach, we coin the term brick to represent each functional module, designating the modularized structure as configurable foundation models. In this paper, we offer a comprehensive overview and investigation of the construction, utilization, and limitation of configurable foundation models. We first formalize modules into emergent bricks - functional neuron partitions that emerge during the pre-training phase, and customized bricks - bricks constructed via additional post-training to improve the capabilities and knowledge of LLMs. Based on diverse functional bricks, we further present four brick-oriented operations: retrieval and routing, merging, updating, and growing. These operations allow for dynamic configuration of LLMs based on instructions to handle complex tasks. To verify our perspective, we conduct an empirical analysis on widely-used LLMs. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions. Finally, we highlight several open issues and directions for future research. Overall, this paper aims to offer a fresh modular perspective on existing LLM research and inspire the future creation of more efficient and scalable foundational models.
Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology of China（中国科学技术大学）
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；College of Design and Innovation, Tongji University（同济大学设计创意学院）
FoMo X: Modular Explainability Signals for Outlier Detection Foundation Models
TU Dortmund University, Dortmund, Germany（图卢兹大学（德累斯顿））；Research Center Trustworthy Data Science and Security, University Alliance Ruhr, Dortmund, Germany（可信数据科学与安全研究中心，鲁尔大学联盟，德累斯顿，德国）
SCNO: Spiking Compositional Neural Operator -- Towards a Neuromorphic Foundation Model for Nuclear PDE Solving
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Indian Institute of Technology Delhi（印度理工学院德里分校）；National Center for Supercomputing Applications（国家超级计算应用中心）
