This paper describes the University of Maryland's submission to the Special Task on Formality Control for Spoken Language Translation at \iwslt, which evaluates translation from English into 6 languages with diverse grammatical formality markers. We investigate to what extent this problem can be addressed with a \textit{single multilingual model}, simultaneously controlling its output for target language and formality. Results show that this strategy can approach the translation quality and formality control achieved by dedicated translation models. However, the nature of the underlying pre-trained language model and of the finetuning samples greatly impact results.
Amazon AWS AI（亚马逊AWS人工智能）；University of Maryland（马里兰大学）
This work aims to produce translations that convey source language content at a formality level that is appropriate for a particular audience. Framing this problem as a neural sequence-to-sequence task ideally requires training triplets consisting of a bilingual sentence pair labeled with target language formality. However, in practice, available training examples are limited to English sentence pairs of different styles, and bilingual parallel sentences of unknown formality. We introduce a novel training scheme for multi-task models that automatically generates synthetic training triplets by …
Huawei Turkey R&D Center（华为土耳其研发中心）；Nisum Latam（Nisum拉美公司）；Banco de Crédito e Inversiones（智利信贷与投资银行）；North South University（南北大学）；Sorbonne Université（索邦大学）
Multilingual generative language models (LMs) are increasingly fluent in a large variety of languages. Trained on the concatenation of corpora in multiple languages, they enable powerful transfer from high-resource languages to low-resource ones. However, it is still unknown what cultural biases are induced in the predictions of these models. In this work, we focus on one language property highly influenced by culture: formality. We analyze the formality distributions of XGLM and BLOOM's predictions, two popular generative multilingual language models, in 5 languages. We classify 1,200 generat…
University of Maryland（马里兰大学）；Dataminr, Inc.（Dataminr公司）
While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading ST automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgm…
University of Maryland（马里兰大学）；Dataminr, Inc.（Dataminr公司）
We take the first step towards multilingual style transfer by creating and releasing XFORMAL, a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on XFORMAL suggest that state-of-the-art style transfer approaches perform close to simple baselines, indicating that style transfer is even more challenging when moving multilingual.
We explore the use of natural language prompts for controlling various aspects of the outputs generated by machine translation models. We demonstrate that natural language prompts allow us to influence properties like formality or specific dialect of the output. We show that using language names to control the output language of multilingual translation models enables positive transfer for unseen language pairs. This unlocks the ability to translate into languages not seen during fine-tuning by using their English names. We investigate how scale, number of pre-training steps, number of languag…
This paper describes the SLT-CDT-UoS group's submission to the first Special Task on Formality Control for Spoken Language Translation, part of the IWSLT 2022 Evaluation Campaign. Our efforts were split between two fronts: data engineering and altering the objective function for best hypothesis selection. We used language-independent methods to extract formal and informal sentence pairs from the provided corpora; using English as a pivot language, we propagated formality annotations to languages treated as zero-shot in the task; we also further improved formality controlling with a hypothesis …
Detecting Text Formality: A Study of Text Classification Approaches
Technical University of Munich（慕尼黑工业大学）；Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS)（智能技术单一研究中心）；Universidade de Santiago de Compostela（圣地亚哥德孔波斯特拉大学）；Skolkovo Institute of Science and Technology（斯科尔科沃科技学院）；Artificial Intelligence Research Institute（人工智能研究所）
We demonstrate the potential of few-shot translation systems, trained with unpaired language data, for both high and low-resource language pairs. We show that with only 5 examples of high-quality translation data shown at inference, a transformer decoder-only model trained solely with self-supervised learning, is able to match specialized supervised state-of-the-art models as well as more general commercial translation systems. In particular, we outperform the best performing system on the WMT'21 English - Chinese news translation task by only using five examples of English - Chinese parallel …
We exploit the pre-trained seq2seq model mBART for multilingual text style transfer. Using machine translated data as well as gold aligned English sentences yields state-of-the-art results in the three target languages we consider. Besides, in view of the general scarcity of parallel data, we propose a modular approach for multilingual formality transfer, which consists of two training strategies that target adaptation to both language and task. Our approach achieves competitive performance without monolingual task-specific parallel data and can be applied to other style transfer tasks as well…
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Center for Language and Cognition, University of Groningen（格罗宁根大学语言与认知中心）
Causal Language Control in Multilingual Transformers via Sparse Feature Steering
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Maryland, College Park（马里兰大学学院公园分校）；Barnard College（巴纳德学院）；Algoverse AI Research（Algoverse人工智能研究）；Meta FAIR Lab（Meta FAIR实验室）
Style transfer is the task of transferring an attribute of a sentence (e.g., formality) while maintaining its semantic content. The key challenge in style transfer is to strike a balance between the competing goals, one to preserve meaning and the other to improve the style transfer accuracy. Prior research has identified that the task of meaning preservation is generally harder to attain and evaluate. This paper proposes two extensions of the state-of-the-art style transfer models aiming at improving the meaning preservation in style transfer. Our evaluation shows that these extensions help t…
In this work, we take the first steps towards building a universal rewriter: a model capable of rewriting text in any language to exhibit a wide variety of attributes, including styles and languages, while preserving as much of the original semantics as possible. In addition to obtaining state-of-the-art results on unsupervised translation, we also demonstrate the ability to do zero-shot sentiment transfer in non-English languages using only English exemplars for sentiment. We then show that our model is able to modify multiple attributes at once, for example adjusting both language and sentim…
CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark
Saarland University（萨尔兰大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Centre for European Research in Trusted AI (CERTAIN)（可信AI欧洲研究中心（CERTAIN））；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；TU Darmstadt（德累斯顿技术大学）；Mila - Quebec Artificial Intelligence Institute（魁北克人工智能研究所（Mila））；McGill University（麦吉尔大学）
Google Research（谷歌研究院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al. 2021) has attempted "few-shot" style transfer using only 3-10 sentences at inference for style extraction. In this work we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim. We push the state-of-the-art for few-shot style tr…
We study politeness phenomena in nine typologically diverse languages. Politeness is an important facet of communication and is sometimes argued to be cultural-specific, yet existing computational linguistic study is limited to English. We create TyDiP, a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples. We evaluate how well multilingual models can identify politeness levels -- they show a fairly robust zero-shot transfer ability, yet fall short of estimated human accuracy significantly. We further study mapping the English politenes…
When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy
University of Groningen（格罗宁根大学）；Harvard University（哈佛大学）；Mass General Brigham（麻省总医院）；Boston Children’s Hospital（波士顿儿童医院）；University of Amsterdam（阿姆斯特丹大学）
Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models
Data & Knowledge Engineering, Heinrich Heine University（海因里希·海因大学数据与知识工程系）；Center for Language and Cognition (CLCG), University of Groningen（格罗宁根大学语言与认知中心）
Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究院）；Stevens Institute of Technology（史蒂文斯理工学院）；Dalhousie University（达尔豪斯大学）
Acoustic word embedding models map variable duration speech segments to fixed dimensional vectors, enabling efficient speech search and discovery. Previous work explored how embeddings can be obtained in zero-resource settings where no labelled data is available in the target language. The current best approach uses transfer learning: a single supervised multilingual model is trained using labelled data from multiple well-resourced languages and then applied to a target zero-resource language (without fine-tuning). However, it is still unclear how the specific choice of training languages affe…
