Importance-based Neuron Allocation for Multilingual Neural Machine Translation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Language and Culture University(北京语言大学)
Multilingual neural machine translation with a single model has drawn much attention due to its capability to deal with multiple languages. However, the current multilingual translation paradigm often makes the model tend to preserve the general knowledge, but ignore the language-specific knowledge. Some previous works try to solve this problem by adding various kinds of language-specific modules to the model, but they suffer from the parameter explosion problem and require specialized manual design. To solve these problems, we propose to divide the model neurons into general and language-specific parts based on their importance across languages. The general part is responsible for preserving the general knowledge and participating in the translation of all the languages, while the language-specific part is responsible for preserving the language-specific knowledge and participating in the translation of some specific languages. Experimental results on several language pairs, covering IWSLT and Europarl corpus datasets, demonstrate the effectiveness and universality of the proposed method.
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Microsoft Research Asia(微软亚洲研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院)
MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Cross-lingual Similarity of Multilingual Representations Revisited
University of Tartu(塔尔图大学);Institute of Computer Science(计算机科学研究所)
相关工作使用CKA和CCA的变体等索引来度量多语言语言模型中跨语言表示的相似性。在本文中,我们认为CKA/CCA的假设与跨语言学习分析的激励目标之一,即,解释zero-shot跨语言迁移。我们强调什么跨语言的相似性,这些指标未能捕捉和提供一个激励性的案例研究\textit{demonstrating the problem empirically}有价值的方面。然后,我们介绍了\textit{Average Neuron-Wise Correlation (ANC)}作为一个简单的替代方案,是免除CKA/CCA的困难,是很好的,特别是在跨语言的背景下。最后,我们使用ANC来构造证据,证明先前引入的“先对齐,然后预测”模式不仅发生在掩蔽语言模型(MLMs)中,而且发生在具有\textit{causal language modeling}目标的多语言模型(CLMs)中。此外,我们表明,该模式扩展到\textit{scaled versions}的MLMs和CLMs(高达85倍的原始mBERT)。\footnote{Our code is publicly available at \url{https://github.com/TartuNLP/xsim}}
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
School of Informatics, Xiamen University(厦门大学信息学院);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室(厦门大学),文化和旅游部)
CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark
Saarland University(萨尔兰大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Centre for European Research in Trusted AI (CERTAIN)(可信AI欧洲研究中心(CERTAIN));University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);TU Darmstadt(德累斯顿技术大学);Mila - Quebec Artificial Intelligence Institute(魁北克人工智能研究所(Mila));McGill University(麦吉尔大学)
One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Zhongguancun Academy(中关村学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tianjin University(天津大学);Nankai University(南开大学)
Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
Liangzhu Laboratory, Zhejiang University(浙江大学良渚实验室);AI Research Center, WeDoctor Cloud(微医云AI研究中心);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);School of Public Health, Zhejiang University(浙江大学公共卫生学院)
MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Tongji University(同济大学)
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
Singapore Management University(新加坡管理学院);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);DAMO Academy, Alibaba Group(阿里集团大模型实验室);Singapore University of Technology and Design(新加坡科技设计大学)