小型语言模型（SLM）的可信性基准测试：预训练模型与压缩模型的对比
Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed
- Institute of Automation, CAS（中国科学院自动化研究所）
- Tsinghua University（清华大学）
- Harvard Medical School（哈佛医学院）
- City University of Hong Kong（香港城市大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究评估SLM的可信性，发现量化压缩预训练模型比剪枝更能保留可信性，且量化压缩可靠大模型得到的SLM比从头训练的小型模型更优，知识蒸馏可进一步提升其可靠性。
AI中文摘要：
小型语言模型（SLM）已成为传统大型语言模型（LLM）的更高效替代方案，在资源受限场景中展现出良好应用潜力。构建SLM的现有方法通常遵循两条路径：从头训练紧凑模型，或使用剪枝、量化、蒸馏等方法压缩更大的预训练模型。随着语言模型日益融入实际应用，确保其可信性已成为关键问题，但如何构建可信的SLM仍是未充分探索的课题。本研究对SLM的可信性进行了多维度综合评估，涵盖公平性、鲁棒性、隐私性和伦理性。我们首先考察了剪枝与量化的影响，发现量化在保留可信性方面比剪枝有效得多。更重要的是，我们证明通过量化压缩可靠的大型模型，相比从头训练小型模型，可生成具有更优可信性和适应性的SLM。此外，从可信的教师模型进行知识蒸馏可进一步提升SLM的可靠性。我们希望这些发现为未来开发和部署可信小型语言模型提供实践指导与基础。
英文摘要：
Small Language Models (SLMs) have emerged as a more efficient alternative to traditional Large Language Models (LLMs), offering promising potential in resource-constrained scenarios. Existing approaches to building SLMs typically follow two paths: training compact models from scratch, or compressing larger pre-trained models using methods such as pruning, quantization, or distillation. As language models become increasingly integrated into real-world applications, ensuring their trustworthiness has become a critical concern. However, how to build trustworthy SLMs remains an underexplored question. In this work, we present a comprehensive evaluation of SLM trustworthiness across multiple dimensions, including fairness, robustness, privacy, and ethics. We first examine the effects of pruning and quantization, and find that quantization is significantly more effective in preserving trustworthiness compared to pruning. More importantly, we demonstrate that compressing a reliable large model via quantization can produce SLMs with superior trustworthiness and adaptability compared to using small models trained from scratch. Furthermore, knowledge distillation from trustworthy teacher models can further enhance the reliability of SLMs. We hope our findings provide practical guidance and a foundation for future research into the development and deployment of trustworthy small language models.
