Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative transfer across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and multi-domain sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains to mitigate their interference. Evaluated in various tasks including speech recognition, text-to-text and speech-to-text translation, the proposed attention sharing strategies consistently bring gains to sequence models built upon multi-head attention. For speech-to-text translation, our approach yields an average of $+2.0$ BLEU over $13$ language directions in multilingual setting and $+2.0$ BLEU over $3$ domains in multi-domain setting.
XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags
Ahsanullah University of Science & Technology(阿萨努拉科技大学);University of Alberta(阿尔伯塔大学);Université de Sherbrooke(谢布鲁克大学);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学)
COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳分校计算机科学与工程系);Department of Metallurgical and Materials Engineering, National Institute of Technology Tiruchirappalli, India(印度理工学院 Tiruchirappalli 金属与材料工程系);Department of Computer Science and Information Systems, BITS Pilani – Goa Campus, India(比斯·帕尼学院 Goa 分校计算机科学与信息系统系);Department of Computer Science and Engineering, Indian Institute of Information Technology Vadodara, India(印度信息科技学院瓦达拉分校计算机科学与工程系);Department of Computer Science and Engineering, B.M.S. College of Engineering, Bangalore, India(班加罗尔 B.M.S. 工程学院计算机科学与工程系);Université de Toulouse, IRIT UMR 5505 CNRS, France(图卢兹大学 IRIT UMR 5505 CNRS 实验室)
Vārta: A Large-Scale Headline-Generation Dataset for Indic Languages
Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);IIT Madras(印度理工学院马德拉斯分校);AI4Bharat(AI4Bharat(印度人工智能研究机构));CIFAR AI Chair(加拿大高等研究院人工智能讲席)
IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages
IIT Madras(印度理工学院马德拉斯分校);Columbia University(哥伦比亚大学);National Institute of Information and Communications Technology(信息通信研究机构);Microsoft(微软);University of Edinburgh(爱丁堡大学);AI4Bharat
Causal Language Control in Multilingual Transformers via Sparse Feature Steering
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Maryland, College Park(马里兰大学学院公园分校);Barnard College(巴纳德学院);Algoverse AI Research(Algoverse人工智能研究);Meta FAIR Lab(Meta FAIR实验室)
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
NII LLMC;Osaka University(大阪大学);Nara Institute of Science and Technology(奈良科学技術大學);University College London(伦敦大学学院);Kyoto University(京都大学);Waseda University(早稻田大学)
Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages
Department of Automation, Tsinghua University, Beijing, China(清华大学自动化系);Alibaba International Digital Commerce Group, Beijing, China(阿里巴巴国际数字 commerce 集团);School of Software, Tsinghua University, Beijing, China(清华大学软件学院)
Towards a resource for multilingual lexicons: an MT assisted and human-in-the-loop multilingual parallel corpus with multi-word expression annotation
Leiden University Medical Center(莱顿大学医学中心);Leiden Institute of Advanced Computer Science (LIACS)(莱顿先进计算机科学研究所);Leiden University(莱顿大学);University of Tours(图尔大学);Arabic Natural Language Processing Research Group(阿拉伯自然语言处理研究组);University of Sfax(索菲亚大学);Institute for Natural Language Processing (IMS)(自然语言处理研究所);University of Stuttgart(斯图加特大学);ADAPT Research Centre(ADAPT研究中心);Dublin City University(都柏林城市大学);Insight Centre for Data Analytics(数据洞察中心);University of Manchester(曼彻斯特大学)
Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑莱茵河大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML));Bosch Center for Artificial Intelligence(博世人工智能中心)
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Microsoft Research(微软研究院)
我们在MiniLM中推广了深度自我注意力蒸馏(Wang et al.,2020)通过仅使用自注意关系蒸馏来对预训练的Transformers进行任务不可知压缩。特别是,我们定义了多头自我注意力的关系,每个自我注意力模块内的查询,键和值向量对之间的缩放点积。然后运用上述相关知识对学生模型进行训练。除了简单和统一的原则,更有利的是,它没有限制的数量方面的学生的注意头,而大多数以前的工作必须保证相同的头数之间的教师和学生。此外,细粒度的自注意关系倾向于充分利用Transformer学习的交互知识。此外,我们彻底研究了教师模型的层选择策略,而不是像MiniLM中那样只依赖最后一层。我们对压缩单语和多语言预训练模型进行了广泛的实验。实验结果表明,我们的模型提取的基础大小和大尺寸的教师(BERT,RoBERTa和XLM-R)优于国家的最先进的。
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Saarland University(萨尔大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);University of Göttingen(哥廷根大学)
Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning
Language Understanding Laboratory, Myanmar(缅甸语言理解实验室);Department of Computer Engineering, KMITL, Bangkok, Thailand(泰国曼谷KMITL计算机工程系);Language and Semantic Technology Research Team, NECTEC, Bangkok, Thailand(泰国曼谷NECTEC语言与语义技术研究团队)