The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative and parallel decoding techniques attempt to mitigate this, they face limitations: either relying on less accurate smaller models for generation or failing to fully leverage the base LLM's representations.
We introduce a novel architecture, Tandem transformers, to address these issues. This architecture uniquely combines (1) a small autoregressive model and (2) a large model operating in block mode (processing multiple tokens simultaneously). The small model's predictive accuracy is substantially enhanced by granting it attention to the large model's richer representations. On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter model with comparable downstream performance. We further incorporate the tandem model within the speculative decoding (SPEED) framework where the large model validates tokens from the small model. This ensures that the Tandem of PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream task accuracy.
Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem
Technische Universität Berlin(柏林工业大学);Saarland Informatics Campus(萨尔兰信息学园区);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Case-Based or Rule-Based: How Do Transformers Do the Math?
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(北京智源人工智能研究院通用人工智能国家重点实验室)
Fusing Large Language Models with Temporal Transformers for Time Series Forecasting
University of Science and Technology of China(中国科学技术大学);University of Washington(华盛顿大学);Beijing Northern Computility InterConnection Co., Ltd.(北京北方计算互联有限公司);ENN Group Co., Ltd.(ENN集团有限公司)
Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers
National University of Singapore(新加坡国立大学);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Integrative Sciences and Engineering Programme, National University of Singapore(新加坡国立大学整合科学与工程计划(注:该计划为新加坡国立大学设立的跨学科项目,此处保留机构相关表述));Tencent(腾讯);Massachusetts Institute of Technology(麻省理工学院);Guangdong Lab of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
LLMCARE: early detection of cognitive impairment via transformer models enhanced by LLM-generated synthetic data
Columbia University Irving Medical Center(哥伦比亚大学欧文医学中心);University of Wisconsin-Milwaukee(威斯康星大学密尔沃基分校);University of Tehran(德黑兰大学);Columbia University(哥伦比亚大学);University of Florida(佛罗里达大学)
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Institute of Trustworthy Embodied Artificial Intelligence, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Pengcheng Laboratory(鹏城实验室);Hikvision Inc(海康威视公司)
Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development
School of Computing and Communications, Lancaster University, UK(兰卡斯特大学计算与通讯学院);College of Engineering & Computer Science, VinUniversity(Vin大学工程与计算机科学学院)
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computing and Mathematical Sciences, Birkbeck, University of London(伦敦大学伯克贝克学院计算与数学科学学院)
Plansformer: Generating Symbolic Plans using Transformers
Artificial Intelligence Institute, University of South Carolina(南卡罗来纳大学人工智能研究所);IBM Research(IBM研究院);University of Parma(帕尔马大学);University of Brescia(布雷西亚大学)
Faith and Fate: Limits of Transformers on Compositionality
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学);University of Chicago(芝加哥大学)
RecurFormer: Not All Transformer Heads Need Self-Attention
Nanyang Technological University(南洋理工大学);The University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);The College of William and Mary(威廉玛丽学院)