arXiv 2402.163672024-10-06cs.CL

解开巴别塔之谜：探索大语言模型的多语言激活模式及其应用

Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications

Liangzhu Laboratory, Zhejiang University（浙江大学良渚实验室）；AI Research Center, WeDoctor Cloud（微医云AI研究中心）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）

Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, Jian Wu

备注：: Accepted to EMNLP 2024 Main Conference

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究将稠密大语言模型转换为细粒度MoE架构，分析其多语言神经元激活模式的异同，并利用激活频率差异指导稀疏激活与剪枝，效果优于随机剪枝，部分语言表现超未剪枝模型，为多语言处理机制研究及模型优化提供新方向。

摘要：

近年来，大语言模型（LLMs）在自然语言处理（NLP）领域取得了巨大突破，但人们对其处理不同语言时的内部神经元活动仍缺乏了解。我们设计了一种将稠密LLMs转换为细粒度MoE（混合专家）架构的方法，随后通过专家激活频率热图可视化研究了LLMs的多语言激活模式。通过在不同模型家族、不同模型规模及不同变体上开展的综合实验，我们分析了LLMs处理不同语言时内部神经元激活模式的异同。具体而言，我们研究了高频激活专家的分布、多语言共享专家、多语言激活模式是否与语系相关，以及指令微调对激活模式的影响。我们进一步探索了利用所发现的专家激活频率差异来指导稀疏激活与剪枝。实验结果表明，我们的方法显著优于随机专家剪枝，在部分语言上的表现甚至超过了未剪枝模型。此外，我们发现根据激活水平差异为不同层配置不同剪枝率可取得更优效果。本研究的发现揭示了LLMs内部的多语言处理机制，并利用这些洞见为稀疏激活、模型剪枝等应用提供了新视角。

英文摘要：

Recently, large language models (LLMs) have achieved tremendous breakthroughs in the field of NLP, but still lack understanding of their internal neuron activities when processing different languages. We designed a method to convert dense LLMs into fine-grained MoE architectures, and then visually studied the multilingual activation patterns of LLMs through expert activation frequency heatmaps. Through comprehensive experiments on different model families, different model sizes, and different variants, we analyzed the similarities and differences in the internal neuron activation patterns of LLMs when processing different languages. Specifically, we investigated the distribution of high-frequency activated experts, multilingual shared experts, whether multilingual activation patterns are related to language families, and the impact of instruction tuning on activation patterns. We further explored leveraging the discovered differences in expert activation frequencies to guide sparse activation and pruning. Experimental results demonstrated that our method significantly outperformed random expert pruning and even exceeded the performance of unpruned models in some languages. Additionally, we found that configuring different pruning rates for different layers based on activation level differences could achieve better results. Our findings reveal the multilingual processing mechanisms within LLMs and utilize these insights to offer new perspectives for applications such as sparse activation and model pruning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。