评估现代大语言模型在低资源和形态丰富的语言上的表现：跨语言基准测试在粤语、日语和土耳其语之间
Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish
AI总结：
本文评估了七种先进LLMs在粤语、日语和土耳其语上的表现，发现专有模型在多语言任务中表现优异，但文化理解和形态泛化仍有不足。
摘要：
大型语言模型（LLMs）在资源丰富的语言如英语上取得了显著成果，但其在低资源和形态丰富的语言上的有效性仍处于探索阶段。本文介绍了对七种最先进的LLMs进行全面评估，包括GPT-4o、GPT-4、Claude~3.5~Sonnet、LLaMA~3.1、Mistral~Large~2、LLaMA-2~Chat~13B和Mistral~7B~Instruct，在一个新的跨语言基准测试中，涵盖粤语、日语和土耳其语。我们的基准测试跨越四个多样化的任务：开放领域问答、文档摘要、英语到X语言翻译和基于文化的对话。我们结合人类评估（评分流畅性、事实准确性、文化适当性）与自动化指标（如BLEU、ROUGE）来评估模型性能。我们的结果表明，尽管最大的专有模型（GPT-4o、GPT-4、Claude~3.5）在语言和任务上普遍领先，但在文化细致理解和形态泛化方面仍存在显著差距。值得注意的是，GPT-4o在跨语言任务上也表现出稳健的多语言性能，而Claude~3.5~Sonnet在知识和推理基准上实现了竞争性的准确性。然而，所有模型在每种语言独特的语言挑战上都面临一定程度的困难，如土耳其语的屈折形态和粤语的口语表达。较小的开源模型（LLaMA-2~13B、Mistral~7B）在流畅性和准确性上显著落后，突显了资源差异。我们提供了详细的定量结果、定性错误分析，并讨论了开发更具文化意识和语言泛化能力的LLMs的含义。我们的基准测试和评估数据已发布，以促进可重复性和进一步研究。
英文摘要：
Large language models (LLMs) have achieved impressive results in high-resource languages like English, yet their effectiveness in low-resource and morphologically rich languages remains underexplored. In this paper, we present a comprehensive evaluation of seven cutting-edge LLMs -- including GPT-4o, GPT-4, Claude~3.5~Sonnet, LLaMA~3.1, Mistral~Large~2, LLaMA-2~Chat~13B, and Mistral~7B~Instruct -- on a new cross-lingual benchmark covering \textbf{Cantonese, Japanese, and Turkish}. Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue. We combine \textbf{human evaluations} (rating fluency, factual accuracy, and cultural appropriateness) with automated metrics (e.g., BLEU, ROUGE) to assess model performance. Our results reveal that while the largest proprietary models (GPT-4o, GPT-4, Claude~3.5) generally lead across languages and tasks, significant gaps persist in culturally nuanced understanding and morphological generalization. Notably, GPT-4o demonstrates robust multilingual performance even on cross-lingual tasks, and Claude~3.5~Sonnet achieves competitive accuracy on knowledge and reasoning benchmarks. However, all models struggle to some extent with the unique linguistic challenges of each language, such as Turkish agglutinative morphology and Cantonese colloquialisms. Smaller open-source models (LLaMA-2~13B, Mistral~7B) lag substantially in fluency and accuracy, highlighting the resource disparity. We provide detailed quantitative results, qualitative error analysis, and discuss implications for developing more culturally aware and linguistically generalizable LLMs. Our benchmark and evaluation data are released to foster reproducibility and further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
