大语言模型时代下的小语言模型综合综述：技术、增强、应用、与LLM的协作及可信性
A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
AI总结：
本文系统综述了小语言模型（SLMs）的定义、获取、应用、增强和可靠性，提出按专门任务能力和资源约束界定SLM，并构建分类法与通用框架，以解决LLMs在隐私、延迟和领域适配上的不足。
摘要：
大语言模型（LLMs）已在文本生成、问答和推理方面展现出涌现能力，促进了各种任务和领域的发展。尽管LLMs在各类任务中表现熟练，但像PaLM 540B和Llama-3.1 405B这样的LLMs由于参数规模庞大和计算需求高而面临局限性，通常需要使用云API，这引发了隐私担忧，限制了边缘设备上的实时应用，并增加了微调成本。此外，LLMs在医疗保健和法律等专业领域往往表现不佳，原因是缺乏领域特定知识，因此需要专门的模型。因此，小语言模型（SLMs）因其低推理延迟、成本效益高、开发高效以及易于定制和适应性而日益受到青睐。这些模型特别适合资源受限的环境和领域知识获取，解决了LLMs面临的挑战，并证明非常适合需要本地化数据处理以保护隐私、最小化推理延迟以提高效率以及通过轻量级微调获取领域知识的应用。对SLMs日益增长的需求推动了广泛的研究和开发。然而，目前仍缺乏一项全面的综述来研究与SLM的定义、获取、应用、增强和可靠性相关的问题，这促使我们对这些主题进行详细调查。SLMs的定义差异很大，因此为了标准化，我们提出根据其执行专门任务的能力和对资源受限环境的适用性来定义SLMs，并基于涌现能力所需的最小规模和资源约束下可持续的最大规模来设定边界。对于其他方面，我们提供了相关模型/方法的分类法，并为每个类别开发了通用框架，以有效地增强和利用SLMs。
英文摘要：
Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
