大型语言模型（LLM）在各类应用中展现出令人瞩目的能力，但也面临高推理延迟、高昂训练成本以及产生幻觉等重大挑战。大型语言模型与小型语言模型（SLM）之间的协作解码通过投机解码、对比解码以及模拟器或代理微调等方法，为缓解这些问题提供了一种颇具前景的策略。然而，此类协作的具体机制，尤其是从统一视角进行的研究，仍在很大程度上未被探索。受双过程认知理论启发，本文提出一个统一框架，称为 Fast and Slow Generating（FS-GEN）。在该框架中，LLM（有时与 SLM 一起）被归类为 System 2（缓慢且审慎），而独立的 SLM 则被指定为 System 1（快速且直觉）。我们对这些协作方法进行了全面分析，阐明其共同属性，并通过 FS-GEN 框架揭示 System 2 相对于 System 1 的差异化知识能力。研究结果表明，在各种方法中，只需要很小比例的协作交互（在大多数情况下约少于20%）。System 1 与 System 2 之间的这些交互遵循与参数比例相关的缩放定律，从而使协作具备可预测性。此外，我们还探讨了协作在何种具体条件下最为有效，尤其从不确定性角度进行分析，提供了可能指导未来优化工作的新见解。我们的研究强调，System 1 与 System 2 之间的根本区别在于下一 token 预测的不确定性，此时 System 2 的干预对于支持 System 1 至关重要。复现代码：https://github.com/TsinghuaC3I/FS-GEN
英文摘要：
Large Language Models (LLMs) exhibit impressive capabilities across various applications but encounter substantial challenges such as high inference latency, considerable training costs, and the generation of hallucinations. Collaborative decoding between large and small language models (SLMs) presents a promising strategy to mitigate these issues through methods including speculative decoding, contrastive decoding, and emulator or proxy fine-tuning. However, the specifics of such collaborations, particularly from a unified perspective, remain largely unexplored. Inspired by dual-process cognitive theory, we propose a unified framework in this paper, termed Fast and Slow Generating (FS-GEN). Within this framework, LLMs (sometimes along with SLMs) are categorized as System 2 (slow and deliberate), while independent SLMs are designated as System 1 (fast and intuitive). We provide a comprehensive analysis of these collaborative methodologies, elucidating their common properties and shedding light on the differential knowledge capabilities of System 2 versus System 1 through the FS-GEN framework. Our findings indicate that only a small proportion of collaborative interactions (approximately less than 20\% in most instances) are necessary across various methods. These interactions between System 1 and System 2 conform to a scaling law related to the parameter ratios, enabling predictable collaboration. Furthermore, we explore the specific conditions under which collaboration proves most effective, particularly from an uncertainty perspective, offering novel insights that may guide future optimization efforts. Our research underscores that the fundamental distinction between System 1 and System 2 lies in the uncertainty of next token predictions, where interventions by System 2 are crucial to support System 1. Code for Reproduction: https://github.com/TsinghuaC3I/FS-GEN
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Intel（英特尔）；Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·拉希德人工智能大学）；Rutgers University（罗格斯大学）
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
Institute of Information Science, Academia Sinica（学术院信息研究所）；Department of Computer Science and Information Engineering, National Taiwan University（台湾大学计算机科学与信息工程系）
Token Level Routing Inference System for Edge Devices
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Rutgers University（罗格斯大学）
CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
A Comparative Study of Decoding Strategies in Medical Text Generation
AI Multimedia Lab（人工智能多媒体实验室）；CAMPUS Research Institute（CAMPUS研究学院）；National University of Science and Technology Politehnica Bucharest（巴尔的效率科学技术大学）；Department of Holistic Systems（整体系统部门）；SimulaMet；Oslo Metropolitan University（奥斯陆 Metropolitan 大学）；Cyber Security（网络安全）；Simula Research Laboratory（Simula研究实验室）
