Recent research has focused on literary machine translation (MT) as a new challenge in MT. However, the evaluation of literary MT remains an open problem. We contribute to this ongoing discussion by introducing LITEVAL-CORPUS, a paragraph-level parallel corpus containing verified human translations and outputs from 9 MT systems, which totals over 2k translations and 13k evaluated sentences across four language pairs, costing 4.5k C. This corpus enables us to (i) examine the consistency and adequacy of human evaluation schemes with various degrees of complexity, (ii) compare evaluations by students and professionals, assess the effectiveness of (iii) LLM-based metrics and (iv) LLMs themselves. Our findings indicate that the adequacy of human evaluation is controlled by two factors: the complexity of the evaluation scheme (more complex is less adequate) and the expertise of evaluators (higher expertise yields more adequate evaluations). For instance, MQM (Multidimensional Quality Metrics), a complex scheme and the de facto standard for non-literary human MT evaluation, is largely inadequate for literary translation evaluation: with student evaluators, nearly 60% of human translations are misjudged as indistinguishable or inferior to machine translations. In contrast, BWS (BEST-WORST SCALING), a much simpler scheme, identifies human translations at a rate of 80-100%. Automatic metrics fare dramatically worse, with rates of at most 20%. Our overall evaluation indicates that published human translations consistently outperform LLM translations, where even the most recent LLMs tend to produce considerably more literal and less diverse translations compared to humans.
LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim（曼海姆大学商业信息学与数学学院自然语言学习组）；University of Aberdeen, Department of Computing Science（阿伯丁大学计算机科学系）；University of Gent, Department of Translation, Interpreting and Communication（根特大学翻译、口译与传播系）；University of Technology Nuremberg (UTN), Department Engineering（纽伦堡技术大学工程系）；Natural Language Learning and Generation (NLLG) Lab（自然语言学习与生成（NLLG）实验室）
Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim（曼海姆大学自然语言学习组（NLLG）商学院信息与数学学院）；University of Technology Nuremberg (UTN), Department Engineering（纽伦堡技术大学（UTN）工程系）；University of Gent, Department of Translation, Interpreting and Communication（根特大学翻译、口译与传播系）；University of Aberdeen, Department of Computing Science（阿伯丁大学计算科学系）；Natural Language Learning and Generation (NLLG) Lab（自然语言学习与生成（NLLG）实验室）
SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation
Beijing Normal University（北京师范大学）；University of Science and Technology of China（中国科学技术大学）；University of Coimbra（科英布拉大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Northwestern Polytechnical University（西北工业大学）
Can ChatGPT Really Understand Modern Chinese Poetry?
NLP；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门科技大学CT实验室）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（人文学院葡萄牙语系，澳门科技大学）
Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish
Centre for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）；Departament de Llenguatges i Sistemes Informàtics, Universitat d’Alacant（语言与信息系统部门，阿尔阿罕特大学）；Departament de Traducció i Comunicació, Universitat Jaume I（翻译与传播部门， Jaime I 大学）
Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translation
CLCG, University of Groningen（格罗宁根大学CLCG）；Department of Computer Science, Aalborg University（奥尔堡大学计算机科学系）；Departament de Llenguatges i Sistemes Informàtics, Universitat d’Alacant（阿莱康特大学语言与信息系统系）
Training BERT Models to Carry Over a Coding System Developed on One Corpus to Another
Pázmány Péter Catholic University（帕兹马尼·彼得天主教大学）；HUN-REN Alfréd Rényi Institute of Mathematics（HUN-REN 阿尔弗雷德·莱尼数学研究所）；Institute of Mathematics, Faculty of Science, Eötvös Loránd University（厄特沃什·罗兰大学理学院数学研究所）
Ancient Greek to Modern Greek Machine Translation: A Novel Benchmark and Fine-Tuning Experiments on LLMs and NMT Models
National and Kapodistrian University of Athens, Department of Informatics and Telecommunications（雅典国家和卡普迪斯特里亚大学信息与电信系）；Institute for Language and Speech Processing, Athena RC（语言与语音处理研究所，雅典RC）
