Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse tasks. We introduce Transformer-Squared, a novel self-adaptation framework that adapts LLMs for unseen tasks in real-time by selectively adjusting only the singular components of their weight matrices. During inference, Transformer-Squared employs a two-pass mechanism: first, a dispatch system identifies the task properties, and then task-specific 'expert' vectors, trained using reinforcement learning, are dynamically mixed to obtain targeted behavior for the incoming prompt. Our method consistently outperforms ubiquitous approaches such as LoRA, with fewer parameters and greater efficiency. Furthermore, Transformer-Squared demonstrates versatility across different LLM architectures and modalities, including vision-language tasks. Transformer-Squared represents a significant leap forward, offering a scalable, efficient solution for enhancing the adaptability and task-specific performance of LLMs, paving the way for truly dynamic, self-organizing AI systems.
Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers
National University of Singapore(新加坡国立大学);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Integrative Sciences and Engineering Programme, National University of Singapore(新加坡国立大学整合科学与工程计划(注:该计划为新加坡国立大学设立的跨学科项目,此处保留机构相关表述));Tencent(腾讯);Massachusetts Institute of Technology(麻省理工学院);Guangdong Lab of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computing and Mathematical Sciences, Birkbeck, University of London(伦敦大学伯克贝克学院计算与数学科学学院)
Faith and Fate: Limits of Transformers on Compositionality
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学);University of Chicago(芝加哥大学)
RecurFormer: Not All Transformer Heads Need Self-Attention
Nanyang Technological University(南洋理工大学);The University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);The College of William and Mary(威廉玛丽学院)
Fusing Large Language Models with Temporal Transformers for Time Series Forecasting
University of Science and Technology of China(中国科学技术大学);University of Washington(华盛顿大学);Beijing Northern Computility InterConnection Co., Ltd.(北京北方计算互联有限公司);ENN Group Co., Ltd.(ENN集团有限公司)
LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
Distributed Operating Systems Group, Department of Informatics, Universität Hamburg, Germany(信息学院分布式操作系统组,乌尔姆大学);Machine Learning Group, Department of Informatics, Universität Hamburg, Germany(信息学院机器学习组,乌尔姆大学)
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);School of EECS, Peking University(电子工程系,北京大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Center for Machine Learning Research, Peking University(机器学习研究中心,北京大学)
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
Middle Tennessee State University(中田纳西州立大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at the University of California, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Case-Based or Rule-Based: How Do Transformers Do the Math?
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(北京智源人工智能研究院通用人工智能国家重点实验室)
A Short Survey of Human Mobility Prediction in Epidemic Modeling from Transformers to LLMs
Groupe de Recherche de Prospection et Valorisation des Données (Greprovad)(数据勘探与增值研究组(Greprovad));GRIC;Université de Sherbrooke(谢布鲁克大学);University of Kinshasa(金沙萨大学);Montreal Behavioural Medicine Centre(蒙特利尔行为医学中心);Centre Intégré Universitaire de Santé et Services Sociaux du Nord-de-l’Île-de-Montréal (CIUSSS-NIM)(蒙特利尔岛北部综合大学卫生与社会服务中心(CIUSSS-NIM));LISV-UVSQ;Université Paris-Saclay(巴黎-萨克雷大学);University of Klagenfurt(克拉根福大学);Biomedical Research Unit, Hospital Monkole(蒙科尔医院生物医学研究单元);University of Florida(佛罗里达大学);School of Electrical Engineering and Computer Science, University of Ottawa(渥太华大学电气工程与计算机科学学院);Centre National d’Intelligence Épidémiologique (CNIEP)(国家流行病学情报中心(CNIEP))
Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development
School of Computing and Communications, Lancaster University, UK(兰卡斯特大学计算与通讯学院);College of Engineering & Computer Science, VinUniversity(Vin大学工程与计算机科学学院)
Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
School of Physics, Peking University(北京大学物理学院);Institute of Theoretical Physics, Chinese Academy of Sciences(中国科学院理论物理研究所);Center for High Energy Physics, Peking University(北京大学高能物理研究中心);Beijing Computational Science Research Center(北京计算科学研究中心)
Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling
School of Computer Science, Peking University(北京大学计算机学院);aiXcoder;The Hong Kong University of Science and Technology(香港科学与技术大学);Advanced Institute of Big Data(大数据高级研究院)
Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Department of Computer Sciences, University of Wisconsin–Madison(威斯康星大学麦迪逊分校计算机科学系);Department of Statistics, University of Wisconsin–Madison(威斯康星大学麦迪逊分校统计学系)