arXiv 2602.241192026-04-15cs.CLcs.AI

评估基于大语言模型的低资源技术语言翻译：加伦的医学与哲学希腊语

Evaluating LLM-Based Translation of a Low-Resource Technical Language: The Medical and Philosophical Greek of Galen

Department of Classical and Mediterranean Studies, Vanderbilt University（维斯尼尔大学古典与地中海研究系）；Center for Hellenic Studies, Harvard University（哈佛大学希腊研究中心）；Department of Philosophy, Vanderbilt University（维斯尼尔大学哲学系）；Machine Intelligence and Normative Theory (MINT) Lab, Australian National University（澳大利亚国立大学机器智能与规范理论实验室）；Department of the Classics, Harvard University（哈佛大学古典学系）

James L. Zainaldin, Cameron Pattison, Manuela Marai, Jacob Wu, Mark J. Schiefsky

备注：: Article + supplementary information

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文评估了商业大语言模型在古希腊技术文本中的机器翻译质量，并将自动化评估指标与专家判断对比，发现术语稀有性是翻译失败的主要预测因素。

摘要：

目的：本研究评估了商业大语言模型（LLM）在古希腊技术文本中的机器翻译（MT）质量，并将标准自动化MT评估指标与专家人类判断进行对比。设计：我们评估了三个LLM（ChatGPT、Claude、Gemini）对加伦（约129-216年）两部作品中20段落长度文本的60种翻译，其中一部为解释性文本包含两个已发表的英文翻译，另一部为从未被翻译过的药理学文本。质量评估使用了七个自动化指标和系统参考自由的人类评估，通过修改的多维质量指标（MQM）框架由领域专家进行评估。发现：在翻译的解释性文本中，LLM表现优异（平均MQM分数95.2/100）。在未翻译的药理学文本中，质量较低（79.9/100），但呈双峰分布：两段文本术语密度极高导致灾难性失败，其余文本评分接近解释性文本。术语稀有性通过语料库频率操作化，成为失败的主要预测因素（r = -0.97）。自动化指标仅在质量差异大的文本中与人类判断有中等相关性；没有指标能区分高质量翻译。原创性：这是首次系统、参考自由的专家人类评估LLM翻译任何古代语言，并且是首次研究识别出预测翻译失败的文本属性。

英文摘要：

Purpose: This study evaluates the quality of commercial large language model (LLM) machine translation (MT) for Ancient Greek technical prose and benchmarks standard automated MT evaluation metrics against expert human judgment. Design: We evaluated 60 translations by three LLMs (ChatGPT, Claude, Gemini) of 20 paragraph-length passages from 2 works by the Greek physician Galen (c. 129-216 CE): an expository text with two published English translations and a pharmacological text never before translated. Quality was assessed using seven automated metrics and systematic reference-free human evaluation via a modified Multidimensional Quality Metrics (MQM) framework applied by domain specialists. Findings: On the translated expository text, LLMs achieved high quality (mean MQM score 95.2/100). On the untranslated pharmacological text, quality was lower (79.9/100) but bimodally distributed: two passages with extreme terminological density produced catastrophic failures, while remaining passages scored within 4 points of the expository text. Terminology rarity, operationalized via corpus frequency, emerged as the dominant predictor of failure (r = -.97). Automated metrics showed moderate correlation with human judgment only on texts with wide quality variance; no metric discriminated among high-quality translations. Originality: This is the first systematic, reference-free expert human evaluation of LLM translation for any ancient language and the first study identifying textual properties predictive of translation failure.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。