This article investigates the performance of automatic evaluation metrics (AEMs) and LLM-as-a-judge evaluation on literary translation across multiple languages, genres, and translation modalities. The aim is to assess how well these tools align with professionals when evaluating translation, creativity (creative shifts & errors), and see if they can substitute laborious manual annotations. A dataset of literary translations across three modalities (human translation, machine translation, and post-editing), three genres and three language pairs was created and annotated in detail for creativity by experienced professional literary translators. The results show that both AEMs and LLM-as-a-judge evaluations correlate poorly with professional evaluations on creativity, with LLM-as-a-judge showing a systematic bias in favour of machine-translated texts and penalising creative and culturally appropriate solutions. Moreover, performance is consistently worse for more literary genres such as poetry. This highlights fundamental limitations of current automatic evaluation tools for literary translation and the need to create new tools that do not frequently consider out of routine translations as errors.
Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish
Centre for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）；Departament de Llenguatges i Sistemes Informàtics, Universitat d’Alacant（语言与信息系统部门，阿尔阿罕特大学）；Departament de Traducció i Comunicació, Universitat Jaume I（翻译与传播部门， Jaime I 大学）
University of Surrey（萨里大学）；University of Groningen（格罗宁根大学）
This article presents the results of a study involving the translation of a short story by Kurt Vonnegut from English to Catalan and Dutch using three modalities: machine-translation (MT), post-editing (PE) and translation without aid (HT). Our aim is to explore creativity, understood to involve novelty and acceptability, from a quantitative perspective. The results show that HT has the highest creativity score, followed by PE, and lastly, MT, and this is unanimous from all reviewers. A neural MT system trained on literary data does not currently have the necessary capabilities for a creative …
University of Surrey（萨里大学）；University of Groningen（格罗宁根大学）
This article presents the results of a study involving the translation of a fictional story from English into Catalan in three modalities: machine-translated (MT), post-edited (MTPE) and translated without aid (HT). Each translation was analysed to evaluate its creativity. Subsequently, a cohort of 88 Catalan participants read the story in a randomly assigned modality and completed a survey. The results show that HT presented a higher creativity score if compared to MTPE and MT. HT also ranked higher in narrative engagement, and translation reception, while MTPE ranked marginally higher in enj…
Evaluating the Creativity of LLMs in Persian Literary Text Generation
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究学院，卡坦大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，德黑兰，伊朗）
Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim（曼海姆大学自然语言学习组（NLLG）商学院信息与数学学院）；University of Technology Nuremberg (UTN), Department Engineering（纽伦堡技术大学（UTN）工程系）；University of Gent, Department of Translation, Interpreting and Communication（根特大学翻译、口译与传播系）；University of Aberdeen, Department of Computing Science（阿伯丁大学计算科学系）；Natural Language Learning and Generation (NLLG) Lab（自然语言学习与生成（NLLG）实验室）
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London（帝国理工学院）；Google DeepMind（谷歌DeepMind）；Institute of Historical Research, School of Advanced Study, University of London（伦敦大学高级研究院历史研究所）；University of Southampton（南安普顿大学）
Universidade do Estado do Rio de Janeiro（里约热内卢州立大学）
In recent years, researchers in the area of Computational Creativity have studied the human creative process proposing different approaches to reproduce it with a formal procedure. In this paper, we introduce a model for the generation of literary rhymes in Spanish, combining structures of language and neural network models %(\textit{Word2vec}).%, into a structure for semantic assimilation. The results obtained with a manual evaluation of the texts generated by our algorithm are encouraging.
SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation
Beijing Normal University（北京师范大学）；University of Science and Technology of China（中国科学技术大学）；University of Coimbra（科英布拉大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Northwestern Polytechnical University（西北工业大学）
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
East China Normal University（华东师范大学）；Shanghai Institute of Artificial Intelligence for Education（上海人工智能教育研究院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Yangzhou University（扬州大学）；School of Education and Intelligent Education Research Center, Yangzhou University（扬州大学教育学院与智能教育研究中心）；School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）
