Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning
Center for Artificial Intelligence Research (CAiRE)（人工智能研究中心（CAiRE））；Department of Electronic and Computer Engineering（电子与计算机工程系）；The Hong Kong University of Science and Technology（香港科技大学）
Zihan Liu, Genta Indra Winata, Andrea Madotto, Pascale Fung
Recently, fine-tuning pre-trained language models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-trained model and weakens its cross-lingual ability, which leads to sub-optimal performance. To alleviate this problem, we leverage continual learning to preserve the original cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. The experimental result shows that our fine-tuning methods can better preserve the cross-lingual ability of the pre-trained model in a sentence retrieval task. Our methods also achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.
Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies
Jožef Stefan Institute（约热夫· Stefan 研究所）；Jožef Stefan International Postgraduate School（约热夫· Stefan 国际研究生学院）；Faculty of Computer and Information Science, University of Ljubljana（卢布尔雅那大学计算机与信息科学学院）
本研究比较大语言模型在跨语言零样本与全样本设置下的 adapter/全参数微调及 IT、CLV 迁移策略，发现 IT 更利于目标语言，CLV 更能保留英语知识。
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Technical University of Munich（慕尼黑工业大学）
Detoxification is a task of generating text in polite style while preserving meaning and fluency of the original toxic text. Existing detoxification methods are designed to work in one exact language. This work investigates multilingual and cross-lingual detoxification and the behavior of large multilingual models like in this setting. Unlike previous works we aim to make large language models able to perform detoxification without direct fine-tuning in given language. Experiments show that multilingual models are capable of performing multilingual style transfer. However, models are not able …
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
Chinese Academy of Sciences（中国科学院）；Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）；Central South University（中南大学）；Langboat Inc.（舟博科技公司）
From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition
School of Computing, Informatics Institute of Technology（计算学院，信息学院技术研究所）；Department of Computer Science & Engineering, University of Moratuwa（计算机科学与工程系，摩图瓦大学）
Improving Cross-lingual Representation for Semantic Retrieval with Code-switching
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Xinjiang Laboratory of Multi-Language Information Technology, Xinjiang University（新疆多语言信息技术实验室）；Xinjiang Multilingual Information Technology Research Center, Xinjiang University（新疆多语信息科技研究中心）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（丝绸之路多语认知计算国际联合实验室）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术学院）；Alibaba DAMO Academy（阿里巴巴达摩院）；Department of Computer Science and Technology, Westlake University（西湖大学计算机科学与技术学院）；Department of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University（北京交通大数据分析与挖掘重点实验室）
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Centre for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Faculty of Mathematics and Physics, Charles University（查理大学数学与物理学院）
Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language
IIT, University of Dhaka（达卡大学理工学院）；Toronto Metropolitan University（多伦多 Metropolitan 大学）；Stony Brook University（石溪大学）；State University of New York at Old Westbury（纽约州立大学奥尔巴尼分校）
Continual Learning (CL) sequentially learns new tasks like human beings, with the goal to achieve better Stability (S, remembering past tasks) and Plasticity (P, adapting to new tasks). Due to the fact that past training data is not available, it is valuable to explore the influence difference on S and P among training examples, which may improve the learning pattern towards better SP. Inspired by Influence Function (IF), we first study example influence via adding perturbation to example weight and computing the influence derivation. To avoid the storage and calculation burden of Hessian inve…
Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学阿利安斯商学院）
Exploring the Efficacy of Federated-Continual Learning Nodes with Attention-Based Classifier for Robust Web Phishing Detection: An Empirical Investigation
School of Computer Science and Engineering（计算机科学与工程学院）；Vellore Institute of Technology（韦洛尔理工学院）
NLP systems typically require support for more than one language. As different languages have different amounts of supervision, cross-lingual transfer benefits languages with little to no training data by transferring from other languages. From an engineering perspective, multilingual NLP benefits development and maintenance by serving multiple languages with a single system. Both cross-lingual transfer and multilingual NLP rely on cross-lingual representations serving as the foundation. As BERT revolutionized representation learning and NLP, it also revolutionized cross-lingual representation…
University of Tokyo（东京大学）；Google Research（谷歌研究院）；Meta AI
While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the modeling of parallel interactions that matters. Exploring this, we examine the usage of unsupervised machine translation to generate synthetic parallel data, and compare it to supervised machine translation and gold parallel data. We find that even model generated parallel data can be useful for downstream tasks, in both a general setting (continued pretraining) as well as the task-specific setting (translate-train)…
University of Cambridge（剑桥大学）；University of Southampton（南安普顿大学）；University of Helsinki（赫尔辛基大学）；HKUST（香港科技大学）
Continual learning approaches help deep neural network models adapt and learn incrementally by trying to solve catastrophic forgetting. However, whether these existing approaches, applied traditionally to image-based tasks, work with the same efficacy to the sequential time series data generated by mobile or embedded sensing systems remains an unanswered question. To address this void, we conduct the first comprehensive empirical study that quantifies the performance of three predominant continual learning schemes (i.e., regularization, replay, and replay with examples) on six datasets from th…
University of Edinburgh（爱丁堡大学）；Institute for Language, Cognition and Computation（语言、认知与计算研究所）
Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English$\leftrightarrow$German, English$\le…
Exploring Open-world Continual Learning with Knowns-Unknowns Knowledge Transfer
Southwestern University of Finance and Economics（西南财经大学）；Leiden University（莱顿大学）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）
Google Research（谷歌研究院）；George Washington University（乔治华盛顿大学）；University of the Basque Country（巴斯克大学）；University of Sheffield（谢菲尔德大学）
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selectio…
ByteDance AI Lab（字节跳动人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Neural machine translation~(NMT) is ineffective for zero-resource languages. Recent works exploring the possibility of unsupervised neural machine translation (UNMT) with only monolingual data can achieve promising results. However, there are still big gaps between UNMT and NMT with parallel supervision. In this work, we introduce a multilingual unsupervised NMT (\method) framework to leverage weakly supervised signals from high-resource language pairs to zero-resource translation directions. More specifically, for unsupervised language pairs \texttt{En-De}, we can make full use of the informa…
City University of Hong Kong（香港城市大学）；Imperial College London（帝国理工学院）
With the capacity of continual learning, humans can continuously acquire knowledge throughout their lifespan. However, computational systems are not, in general, capable of learning tasks sequentially. This long-standing challenge for deep neural networks (DNNs) is called catastrophic forgetting. Multiple solutions have been proposed to overcome this limitation. This paper makes an in-depth evaluation of the memory replay methods, exploring the efficiency, performance, and scalability of various sampling strategies when selecting replay data. All experiments are conducted on multiple datasets …
King’s College London（伦敦国王学院）；MIT（麻省理工学院）；Alan Turing Institute（艾伦·图灵研究所）；Queen Mary University of London（伦敦玛丽女王大学）
Social media has been on the vanguard of political information diffusion in the 21st century. Most studies that look into disinformation, political influence and fake-news focus on mainstream social media platforms. This has inevitably made English an important factor in our current understanding of political activity on social media. As a result, there has only been a limited number of studies into a large portion of the world, including the largest, multilingual and multi-cultural democracy: India. In this paper we present our characterisation of a multilingual social network in India called…
Catastrophic forgetting describes the fact that machine learning models will likely forget the knowledge of previously learned tasks after the learning process of a new one. It is a vital problem in the continual learning scenario and recently has attracted tremendous concern across different communities. In this paper, we explore the catastrophic forgetting phenomena in the context of quantum machine learning. We find that, similar to those classical learning models based on neural networks, quantum learning systems likewise suffer from such forgetting problem in classification tasks emerging…
Both few-shot and continual learning have seen substantial progress in the last years due to the introduction of proper benchmarks. That being said, the field has still to frame a suite of benchmarks for the highly desirable setting of continual few-shot learning, where the learner is presented a number of few-shot tasks, one after the other, and then asked to perform well on a validation set stemming from all previously seen tasks. Continual few-shot learning has a small computational footprint and is thus an excellent setting for efficient investigation and experimentation. In this paper we …
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Chinese Academy of Sciences（中国科学院）；Tianjin University（天津大学）
The backpropagation networks are notably susceptible to catastrophic forgetting, where networks tend to forget previously learned skills upon learning new ones. To address such the 'sensitivity-stability' dilemma, most previous efforts have been contributed to minimizing the empirical risk with different parameter regularization terms and episodic memory, but rarely exploring the usages of the weight loss landscape. In this paper, we investigate the relationship between the weight loss landscape and sensitivity-stability in the continual learning scenario, based on which, we propose a novel me…
