Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.
Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；University College London（伦敦大学学院）；University of Cambridge（剑桥大学）；LanBridge Group（兰桥集团）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究院）
Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish
Centre for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）；Departament de Llenguatges i Sistemes Informàtics, Universitat d’Alacant（语言与信息系统部门，阿尔阿罕特大学）；Departament de Traducció i Comunicació, Universitat Jaume I（翻译与传播部门， Jaime I 大学）
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Nanjing University（南京大学）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
How Well Do Large Reasoning Models Translate? A Comprehensive Evaluation for Multi-Domain Machine Translation
Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能学院）；School of Informatics, Xiamen University（厦门大学信息学院）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University),Ministry of Culture and Tourism（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室（厦门大学），文化部）
Center for Speech and Language Processing（语音与语言处理中心）；Department of Computer Science（计算机科学系）；Johns Hopkins University（约翰斯·霍普金斯大学）
Draft of textbook chapter on neural machine translation. a comprehensive treatment of the topic, ranging from introduction to neural networks, computation graphs, description of the currently dominant attentional sequence-to-sequence model, recent refinements, alternative architectures and challenges. Written as chapter for the textbook Statistical Machine Translation. Used in the JHU Fall 2017 class on machine translation.
National Institute of Information and Communications Technology（日本信息通信技术研究所）
In neural machine translation (NMT), monolingual data in the target language are usually exploited through a method so-called "back-translation" to synthesize additional training parallel data. The synthetic data have been shown helpful to train better NMT, especially for low-resource language pairs and domains. Nonetheless, large monolingual data in the target domains or languages are not always available to generate large synthetic parallel data. In this work, we propose a new method to generate large synthetic parallel data leveraging very small monolingual data in a specific domain. We fin…
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
Digital Twin Institute, Eastern Institute of Technology, Ningbo, China（数字孪生研究所，东部技术研究所，宁波，中国）；Saarland University（萨尔兰州大学）；Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
On Creating an English-Thai Code-switched Machine Translation in Medical Domain
Bangkok Christian International School（曼谷基督教国际学校）；Vidyasirimedhi Institute of Science and Technology（维迪亚西密提科技学院）；Chulalongkorn University（朱拉隆功大学）；Mahidol University（玛希隆大学）
We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: few-shot amplification, distillation, and backtranslation. We first use the zero-shot translation ability of large pre-trained language models to generate translations for a small set of unlabeled sentences. We then amplify these zero-shot translations by using them as few-shot demonstrations for sampling a larger synthetic dataset. This dataset is distilled by discarding the few-shot demonstrations and then fine-tuning. Durin…
National Institute of Information and Communications Technology (NICT)（国立信息与通信技术研究所）；Osaka University（大阪大学）；Microsoft AI and Research（微软人工智能与研究院）
We present a survey on multilingual neural machine translation (MNMT), which has gained a lot of traction in the recent years. MNMT has been useful in improving translation quality as a result of translation knowledge transfer (transfer learning). MNMT is more promising and interesting than its statistical machine translation counterpart because end-to-end modeling and distributed representations open new avenues for research on machine translation. Many approaches have been proposed in order to exploit multilingual parallel corpora for improving translation quality. However, the lack of a com…
College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
Grammar error correction (GEC) is an important application aspect of natural language processing techniques. The past decade has witnessed significant progress achieved in GEC for the sake of increasing popularity of machine learning and deep learning, especially in late 2010s when near human-level GEC systems are available. However, there is no prior work focusing on the whole recapitulation of the progress. We present the first survey in GEC for a comprehensive retrospect of the literature in this area. We first give the introduction of five public datasets, data annotation schema, two impor…
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；LUMLIS (Lisbon ELLIS Unit)（LUMLIS（里斯本ELLIS单元））；University of Edinburgh（爱丁堡大学）；Unbabel（Unbabel公司）
Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous work has been limited in several ways: it often resorts to artificial settings where the problem is amplified, it disregards some (common) types of hallucinations, and it does not validate adequacy of detection heuristics. In this paper, we set foundations for the study of NMT hallucinations. First, we work in a natural setting, i.e., in-domain data without artificial noise neither in training nor in inference. Next…
ByteDance AI Lab（字节跳动人工智能实验室）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pre-trained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and paralle…
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing, Fudan University（复旦大学上海智能视觉计算协同创新中心）
The GPT-3.5 models have demonstrated impressive performance in various Natural Language Processing (NLP) tasks, showcasing their strong understanding and reasoning capabilities. However, their robustness and abilities to handle various complexities of the open world have yet to be explored, which is especially crucial in assessing the stability of models and is a key aspect of trustworthy AI. In this study, we perform a comprehensive experimental analysis of GPT-3.5, exploring its robustness using 21 datasets (about 116K test samples) with 66 text transformations from TextFlint that cover 9 po…
Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation
National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"（乌克兰国家技术大学 "伊戈尔·西科斯基基土夫技术学院"）；Ukrainian Catholic University（乌克兰天主教大学）
Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
AI Agent Lab, Vokram Group（AI代理实验室，Vokram集团）；Purdue University（普渡大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Minnesota（明尼苏达大学）；Emory University（埃默里大学）；Imperial College London（伦敦帝国学院）
