多语言脑外科医生：大型语言模型可被压缩而不遗漏任何语言
Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind
AI总结：
本文提出多语言脑外科医生（MBS）校准数据采样方法，按训练集语言分布比例采样多语言数据，以改善多语言LLM压缩中低资源语言性能下降的问题，并在BLOOM上验证了其有效性。
摘要：
大型语言模型（LLMs）开启了自然语言处理的新纪元，但其庞大的规模要求有效的压缩技术以实现实际应用。尽管已有大量模型压缩技术被研究，它们通常依赖一个忽略多语言语境的校准集，导致低资源语言的准确率显著下降。本文提出了多语言脑外科医生（Multilingual Brain Surgeon，MBS），一种用于多语言LLM压缩的新型校准数据采样方法。MBS通过按模型训练数据集的语言分布比例从多种语言中采样校准数据，克服了现有方法以英语为中心的局限性。我们在BLOOM多语言LLM上进行的实验表明，MBS提升了现有以英语为中心的压缩方法的性能，尤其对低资源语言效果显著。我们还揭示了压缩过程中语言相互作用的动态规律：一种语言在训练集中所占比例越大，且与校准语言越相似，该语言在压缩后保留的性能就越好。总之，MBS提出了一种压缩多语言LLM的创新方法，解决了性能差异问题，并提升了现有压缩技术的语言包容性。
英文摘要：
Large Language Models (LLMs) have ushered in a new era in Natural Language Processing, but their massive size demands effective compression techniques for practicality. Although numerous model compression techniques have been investigated, they typically rely on a calibration set that overlooks the multilingual context and results in significant accuracy degradation for low-resource languages. This paper introduces Multilingual Brain Surgeon (MBS), a novel calibration data sampling method for multilingual LLMs compression. MBS overcomes the English-centric limitations of existing methods by sampling calibration data from various languages proportionally to the language distribution of the model training datasets. Our experiments, conducted on the BLOOM multilingual LLM, demonstrate that MBS improves the performance of existing English-centric compression methods, especially for low-resource languages. We also uncover the dynamics of language interaction during compression, revealing that the larger the proportion of a language in the training set and the more similar the language is to the calibration language, the better performance the language retains after compression. In conclusion, MBS presents an innovative approach to compressing multilingual LLMs, addressing the performance disparities and improving the language inclusivity of existing compression techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
