Encoder-decoder has been widely used in neural machine translation (NMT). A few methods have been proposed to improve it with multiple passes of decoding. However, their full potential is limited by a lack of appropriate termination policies. To address this issue, we present a novel architecture, Rewriter-Evaluator. It consists of a rewriter and an evaluator. Translating a source sentence involves multiple passes. At every pass, the rewriter produces a new translation to improve the past translation and the evaluator estimates the translation quality to decide whether to terminate the rewriting process. We also propose prioritized gradient descent (PGD) that facilitates training the rewriter and the evaluator jointly. Though incurring multiple passes of decoding, Rewriter-Evaluator with the proposed PGD method can be trained with a similar time to that of training encoder-decoder models. We apply the proposed architecture to improve the general NMT models (e.g., Transformer). We conduct extensive experiments on two translation tasks, Chinese-English and English-German, and show that the proposed architecture notably improves the performances of NMT models and significantly outperforms previous baselines.
Although recent years have brought significant progress in improving translation of unambiguously gendered sentences, translation of ambiguously gendered input remains relatively unexplored. When source gender is ambiguous, machine translation models typically default to stereotypical gender roles, perpetuating harmful bias. Recent work has led to the development of "gender rewriters" that generate alternative gender translations on such ambiguous inputs, but such systems are plagued by poor linguistic coverage. To encourage better performance on this task we present and release GATE, a lingui…
Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
NLP（自然语言处理）；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Obtaining meaningful quality scores for machine translation systems through human evaluation remains a challenge given the high variability between human evaluators, partly due to subjective expectations for translation quality for different language pairs. We propose a new metric called XSTS that is more focused on semantic equivalence and a cross-lingual calibration method that enables more consistent assessment. We demonstrate the effectiveness of these novel contributions in large scale evaluation studies across up to 14 language pairs, with translation both into and out of English.
For the past 60 years, Research in machine translation is going on. For the development in this field, a lot of new techniques are being developed each day. As a result, we have witnessed development of many automatic machine translators. A manager of machine translation development project needs to know the performance increase/decrease, after changes have been done in his system. Due to this reason, a need for evaluation of machine translation systems was felt. In this article, we shall present the evaluation of some machine translators. This evaluation will be done by a human evaluator and …
INESC-ID Lisboa（里斯本系统与计算机工程研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Unbabel AI（Unbabel人工智能）；Centro de Linguística da Universidade de Lisboa（里斯本大学语言学中心）；Faculdade de Letras da Universidade de Lisboa（里斯本大学文学院）
In Machine Translation, assessing the quality of a large amount of automatic translations can be challenging. Automatic metrics are not reliable when it comes to high performing systems. In addition, resorting to human evaluators can be expensive, especially when evaluating multiple systems. To overcome the latter challenge, we propose a novel application of online learning that, given an ensemble of Machine Translation systems, dynamically converges to the best systems, by taking advantage of the human feedback available. Our experiments on WMT'19 datasets show that our online approach quickl…
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether the different training methods result in systematically different output beyond what is visible via quality metrics like adequacy or BLEU. We compare translations from supervised and unsupervised MT systems of similar quality, finding that unsupervised output is more fluent and more structurally different in comparison to human translation than is supervised M…
Center for Language and Cognition（语言与认知中心）；University of Groningen（格罗宁根大学）
We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human …
University of Groningen（格罗宁根大学）；Dublin City University（都柏林城市大学）
We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that huma…
Pronouns are important determinants of a text's meaning but difficult to translate. This is because pronoun choice can depend on entities described in previous sentences, and in some languages pronouns may be dropped when the referent is inferrable from the context. These issues can lead Neural Machine Translation (NMT) systems to make critical errors on pronouns that impair intelligibility and even reinforce gender bias. We investigate the severity of this pronoun issue, showing that (1) in some domains, pronoun choice can account for more than half of a NMT systems' errors, and (2) pronouns …
This paper describes the Microsoft Translator submissions to the WMT19 news translation shared task for English-German. Our main focus is document-level neural machine translation with deep transformer models. We start with strong sentence-level baselines, trained on large-scale data created via data-filtering and noisy back-translation and find that back-translation seems to mainly help with translationese input. We explore fine-tuning techniques, deeper models and different ensembling strategies to counter these effects. Using document boundaries present in the authentic and synthetic parall…
Institute of Natural Language Processing, University of Stuttgart（斯图加特大学自然语言处理研究所）；Konkuk University（建国大学）
Natural Language Generation (NLG) refers to the operation of expressing the calculation results of a system in human language. Since the quality of generated sentences from an NLG model cannot be fully represented using only quantitative evaluation, they are evaluated using qualitative evaluation by humans in which the meaning or grammar of a sentence is scored according to a subjective criterion. Nevertheless, the existing evaluation methods have a problem as a large score deviation occurs depending on the criteria of evaluators. In this paper, we propose Grammar Accuracy Evaluation (GAE) tha…
This paper elaborates on the design of a machine translation evaluation method that aims to determine to what degree the meaning of an original text is preserved in translation, without looking into the grammatical correctness of its constituent sentences. The basic idea is to have a human evaluator take the sentences of the translated text and, for each of these sentences, determine the semantic relationship that exists between it and the sentence immediately preceding it. In order to minimise evaluator dependence, relations between sentences are expressed in terms of the conjuncts that can c…
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；ByteDance AI Lab（字节跳动人工智能实验室）；Collaborative Innovation Center of Novel Software Technology and Industrialization（现代软件技术与产业化协同创新中心）
As patients' access to their doctors' clinical notes becomes common, translating professional, clinical jargon to layperson-understandable language is essential to improve patient-clinician communication. Such translation yields better clinical outcomes by enhancing patients' understanding of their own health conditions, and thus improving patients' involvement in their own care. Existing research has used dictionary-based word replacement or definition insertion to approach the need. However, these methods are limited by expert curation, which is hard to scale and has trouble generalizing to …
Early-Exit and Instant Confidence Translation Quality Estimation
ETH Zurich（苏黎世联邦理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；University of Cambridge（剑桥大学）
MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators
College of Computing and Data Science at Nanyang Technological University（南洋理工大学计算与数据科学学院）；Southeast University（东南大学）；The University of Sydney（悉尼大学）；Southeast University Shenzhen Research Institute（东南大学深圳研究院）
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
Sharif University of Technology（谢里夫理工学院）；Vellore Institute of Technology（韦洛雷理工学院）；IIT Kharagpur（印度理工学院达卡分校）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；School of Foreign Languages, East China University of Science and Technology（华东理工大学外国语学院）；Mejiro University（目白大学）
DEEP: Docker-based Execution and Evaluation Platform
PRHLT Research Center - Universitat Politècnica de València（PRHLT研究中心 - 工业技术理工大学）；ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence（ValgrAI - 巴伦西亚人工智能研究生学院和研究网络）
Over the past decade, large-scale supervised learning corpora have enabled machine learning researchers to make substantial advances. However, to this date, there are no large-scale question-answer corpora available. In this paper we present the 30M Factoid Question-Answer Corpus, an enormous question answer pair corpus produced by applying a novel neural network architecture on the knowledge base Freebase to transduce facts into natural language questions. The produced question answer pairs are evaluated both by human evaluators and using automatic evaluation metrics, including well-establish…
Fine-grained Gender Control in Machine Translation with Large Language Models
LG AI Research（LG AI研究院）；Seoul National University（首尔大学）；IPAI, Seoul National University（首尔大学IPAI）；Institute of Engineering Research, Seoul National University（首尔大学工程研究所）
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
Southeast University（东南大学）；Nanjing University（南京大学）；The University of Sydney（悉尼大学）；Shenzhen Research Institute（深圳研究院）；Microsoft（微软公司）；Nanyang Technological University（南洋理工大学）
