The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations. We reassess Hassan et al.'s 2018 investigation into Chinese to English news translation, showing that the finding of human-machine parity was owed to weaknesses in the evaluation design - which is currently considered best practice in the field. We show that the professional human translations contained significantly fewer errors, and that perceived quality in human evaluation depends on the choice of raters, the availability of linguistic context, and the creation of reference translations. Our results call for revisiting current best practices to assess strong machine translation systems in general and human-machine parity in particular, for which we offer a set of recommendations based on our empirical findings.
University of Zurich（苏黎世大学）；University of Edinburgh（爱丁堡大学）
Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task. We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents. In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. Our findings emphasise the need to shift towards document-level evaluation as machine translation im…
Center for Language and Cognition（语言与认知中心）；University of Groningen（格罗宁根大学）
We reassess the claims of human parity and super-human performance made at the news shared task of WMT 2019 for three translation directions: English-to-German, English-to-Russian and German-to-English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersentential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference translation. We then conduct a modified evaluation taking these issues into account. Our results indicate that all the claims of human parity and super-human …
University of Groningen（格罗宁根大学）；Dublin City University（都柏林城市大学）
We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that huma…
Instructor-Worker Large Language Model System for Policy Recommendation: a Case Study on Air Quality Analysis of the January 2025 Los Angeles Wildfires
Department of Systems Design Engineering（系统设计工程系）；College of Land Engineering（土地工程学院）；School of Computer Science（计算机科学学院）；Department of Geography and Environmental Management（地理与环境管理系）；Department of Geomatics Engineering（测绘工程系）
University of Chicago（芝加哥大学）；Deptartment of Computer Science（计算机科学系）；Department of Linguistics（语言学系）
Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases. However, we have limited understanding of how these models handle representation of phrases, and whether this reflects sophisticated composition of phrase meaning like that done by humans. In this paper, we present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word o…
Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French
University of Exeter（埃克塞特大学）；France Éducation international（法国教育国际）；Cental, IL&C, UCLouvain；Computer Science and Engineering Department, Universidad Carlos III de Madrid（马德里卡斯蒂利亚大学计算机科学与工程系）
Carnegie Mellon University（卡内基梅隆大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Indiana University（印第安纳大学）
Although approximately 50% of medical school graduates today are women, female physicians tend to be underrepresented in senior positions, make less money than their male counterparts and receive fewer promotions. There is a growing body of literature demonstrating gender bias in various forms of evaluation in medicine, but this work was mainly conducted by looking for specific words using fixed dictionaries such as LIWC and focused on recommendation letters. We use a dataset of written and quantitative assessments of medical student performance on individual shifts of work, collected across m…
Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs
Erasmus Mundus Master in Language and Communication Technologies（埃拉斯谟 Mundus 语言与沟通技术硕士项目）；HiTZ Center - Ixa, University of the Basque Country (UPV/EHU)（巴斯克国家大学（UPV/EHU）HiTZ中心 - Ixa）；Cohere（Cohere公司）；Saarland University（萨尔兰州大学）
Assessing socio-economic climate impacts from text data
Helmholtz Centre for Environmental Research, Germany（德国海德堡环境研究中心）；Leipzig University, Germany（德国莱比锡大学）；UCLouvain Brussels, Belgium（比利时布鲁塞尔UCLouvain）；Ecole des Ponts, France（法国École des Ponts）；Université PSL, École Polytechnique, Institut Polytechnique de Paris, Sorbonne Université, CNRS, France（法国巴黎理工学院、巴黎理工研究院、巴黎政治学院、法国国家科学研究中心）；RISE Research Institutes of Sweden, Sweden（瑞典RISE研究机构）；Swedish Centre for Impacts of Climate Extremes (Climes), Sweden（瑞典气候极端影响研究中心）；Vrije Universiteit Brussel, Belgium（比利时布鲁塞尔自由大学）；TUD Dresden University of Technology, Germany（德国德累斯顿理工大学）；Uppsala University, Sweden（瑞典乌普萨拉大学）；Politecnico di Milano, Italy（意大利米兰理工大学）；Istituto Universitario di Studi Superiori IUSS Pavia, Italy（意大利皮亚维亚大学研究院）；Center for Climate Change and Transformation, Eurac Research, Italy（意大利Eurac研究机构气候变化与转型中心）；Royal Museum for Central Africa, Belgium（比利时皇家中非博物馆）；Technopolis Group, Germany（德国Technopolis集团）
