While alignment of texts on the sentential level is often seen as being too coarse, and word alignment as being too fine-grained, bi- or multilingual texts which are aligned on a level in-between are a useful resource for many purposes. Starting from a number of examples of non-literal translations, which tend to make alignment difficult, we describe an alignment model which copes with these cases by explicitly coding them. The model is based on predicate-argument structures and thus covers the middle ground between sentence and word alignment. The model is currently used in a recently initiated project of a parallel English-German treebank (FuSe), which can in principle be extended with additional languages.
AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；Key Laboratory of AI Safety, Chinese Academy of Sciences（人工智能安全重点实验室，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Baidu Inc.（百度公司）
University of Chicago（芝加哥大学）；University of Washington（华盛顿大学）
This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations. We thus introduce dual-pivot transfer: training on one language pair and evaluating on other pairs. To study this theory, we design unsupervised models trained on unpaired sentences and single-pair supervised models trained on bitexts, both based on the unsupervised language model XLM-R wit…
University of Stuttgart（斯图加特大学）；VinAI Research（VinAI研究院）；EPFL（洛桑联邦理工学院）；Griffith University（格里菲斯大学）
Knowledge graph (KG) alignment and completion are usually treated as two independent tasks. While recent work has leveraged entity and relation alignments from multiple KGs, such as alignments between multilingual KGs with common entities and relations, a deeper understanding of the ways in which multilingual KG completion (MKGC) can aid the creation of multilingual KG alignments (MKGA) is still limited. Motivated by the observation that structural inconsistencies -- the main challenge for MKGA models -- can be mitigated through KG completion methods, we propose a novel model for jointly compl…
MPO: Multilingual Safety Alignment via Reward Gap Optimization
Harbin Institute of Technology（哈尔滨工业大学）；Singapore Management University（新加坡管理学院）；Monash University（墨尔本大学）；Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Department of Linguistics and Translation, City University of Hong Kong（香港城市大学语言学及翻译学系）
Multilingual sentence representations pose a great advantage for low-resource languages that do not have enough data to build monolingual models on their own. These multilingual sentence representations have been separately exploited by few research for document and sentence alignment. However, most of the low-resource languages are under-represented in these pre-trained models. Thus, in the context of low-resource languages, these models have to be fine-tuned for the task at hand, using additional data sources. This paper presents a weighting mechanism that makes use of available small-scale …
Document alignment techniques based on multilingual sentence representations have recently shown state of the art results. However, these techniques rely on unsupervised distance measurement techniques, which cannot be fined-tuned to the task at hand. In this paper, instead of these unsupervised distance measurement techniques, we employ Metric Learning to derive task-specific distance measurements. These measurements are supervised, meaning that the distance measurement metric is trained using a parallel dataset. Using a dataset belonging to English, Sinhala, and Tamil, which belong to three …
Department of Computer Science（计算机科学系）；Johns Hopkins University（约翰斯·霍普金斯大学）
Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bitexts are unavailable for many languages. In this paper, we propose a new contrastive alignment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sample…
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
NTT Communication Science Laboratories, NTT Corporation（日本电信电话株式会社NTT通信科学实验室）
We present a novel supervised word alignment method based on cross-language span prediction. We first formalize a word alignment problem as a collection of independent predictions from a token in the source sentence to a span in the target sentence. As this is equivalent to a SQuAD v2.0 style question answering task, we then solve this problem by using multilingual BERT, which is fine-tuned on a manually created gold word alignment data. We greatly improved the word alignment accuracy by adding the context of the token to the question. In the experiments using five word alignment datasets amon…
The Hong Kong University of Science and Technology（香港科技大学）；EMOS Technologies Inc.（EMOS科技公司）
We propose a multilingual model to recognize Big Five Personality traits from text data in four different languages: English, Spanish, Dutch and Italian. Our analysis shows that words having a similar semantic meaning in different languages do not necessarily correspond to the same personality traits. Therefore, we propose a personality alignment method, GlobalTrait, which has a mapping for each trait from the source language to the target language (English), such that words that correlate positively to each trait are close together in the multilingual vector space. Using these aligned embeddi…
CM-Align: Consistency-based Multilingual Alignment for Large Language Models
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing Jiaotong University, Ministry of Education（大数据与人工智能交通运输 key laboratory，北京交通大学，教育部）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（模式识别中心，微信AI，腾讯公司，中国）
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Amazon Alexa（亚马逊 Alexa）
Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not pre-trained with explicit cross-lingual supervision, transfer performance can further be improved by aligning mBERT with cross-lingual signal. Prior work proposes several approaches to align contextualised embeddings. In this paper we analyse how different forms of cross-lingual supervision and various alignment methods influence the transfer capability of mBERT in zero-shot setting. Specifically, we compare parallel corpora vs. dictionary-based…
Shanghai University of Finance and Economics（上海财经大学）；Southern University of Science and Technology（南方科技大学）
Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction. However, these methods usually start from mBERT or XLM-R. In this paper, we investigate whether multilingual sentence Transformer LaBSE is a strong multilingual word aligner. This idea is non-trivial as LaBSE is trained to learn language-agnostic sentence-level embeddings, while the alignment extraction task requires the more fine-grained word-level embeddings to be language-agnostic. We demonstrate that the vanilla LaBSE outperforms other mPLMs currently used in the alignme…
Human fixation patterns have been shown to correlate strongly with Transformer-based attention. Those correlation analyses are usually carried out without taking into account individual differences between participants and are mostly done on monolingual datasets making it difficult to generalise findings. In this paper, we analyse eye-tracking data from speakers of 13 different languages reading both in their native language (L1) and in English as language learners (L2). We find considerable differences between languages but also that individual reading behaviour such as skipping rate, total r…
Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
Institute for Analytics and Data Science, University of Essex（数据分析与科学研究所，埃塞克斯大学）；School of Computer Science and Electronic Engineering, University of Essex（计算机科学与电子工程学院，埃塞克斯大学）
Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Multilingual models jointly pretrained on multiple languages have achieved remarkable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to generalize to unseen languages. In this paper, we first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning…
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；National University of Singapore（新加坡国立大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Some Transformer-based models can perform cross-lingual transfer learning: those models can be trained on a specific task in one language and give relatively good results on the same task in another language, despite having been pre-trained on monolingual tasks only. But, there is no consensus yet on whether those transformer-based models learn universal patterns across languages. We propose a word-level task-agnostic method to evaluate the alignment of contextualized representations built by such models. We show that our method provides more accurate translated word pairs than previous method…
We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting a…
University of California, Berkeley（加州大学伯克利分校）；Computer Science Division（计算机科学系）
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task…
Indian Institute of Technology Hyderabad（印度海得拉巴理工学院）
Due to the wide adoption of social media platforms like Facebook, Twitter, etc., there is an emerging need of detecting online posts that can go against the community acceptance standards. The hostility detection task has been well explored for resource-rich languages like English, but is unexplored for resource-constrained languages like Hindidue to the unavailability of large suitable data. We view this hostility detection as a multi-label multi-class classification problem. We propose an effective neural network-based technique for hostility detection in Hindi posts. We leverage pre-trained…
University of Pennsylvania（宾夕法尼亚大学）；IBM Research（IBM研究院）
We develop high performance multilingualAbstract Meaning Representation (AMR) sys-tems by projecting English AMR annotationsto other languages with weak supervision. Weachieve this goal by bootstrapping transformer-based multilingual word embeddings, in partic-ular those from cross-lingual RoBERTa (XLM-R large). We develop a novel technique forforeign-text-to-English AMR alignment, usingthe contextual word alignment between En-glish and foreign language tokens. This wordalignment is weakly supervised and relies onthe contextualized XLM-R word embeddings.We achieve a highly competitive performa…
Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. Recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level repre…
Detecting fine-grained differences in content conveyed in different languages matters for cross-lingual NLP and multilingual corpora analysis, but it is a challenging machine learning problem since annotation is expensive and hard to scale. This work improves the prediction and annotation of fine-grained semantic divergences. We introduce a training strategy for multilingual BERT models by learning to rank synthetic divergent examples of varying granularity. We evaluate our models on the Rationalized English-French Semantic Divergences, a new dataset released with this work, consisting of Engl…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Amazon.com Inc（亚马逊公司）
Predicting missing facts in a knowledge graph (KG) is crucial as modern KGs are far from complete. Due to labor-intensive human labeling, this phenomenon deteriorates when handling knowledge represented in various languages. In this paper, we explore multilingual KG completion, which leverages limited seed alignment as a bridge, to embrace the collective knowledge from multiple languages. However, language alignment used in prior works is still not fully exploited: (1) alignment pairs are treated equally to maximally push parallel entities to be close, which ignores KG capacity inconsistency; …
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
School of Informatics, Xiamen University（厦门大学信息学院）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能学院）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室（厦门大学），文化和旅游部）
Language Model Alignment in Multilingual Trolley Problems
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）；Allen Institute for AI (AI2)（人工智能研究所）；Carnegie Mellon University（卡内基梅隆大学）；University of Trieste（特里斯坦大学）；University of Michigan（密歇根大学）
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
Information and Computer Science Department, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油和矿物大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
