超越理解：评估LLMs在处理隐喻语言中的文化差距
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language
AI总结：
本文评估了LLMs在处理隐喻语言中的文化理解差距，发现其在语用使用方面存在显著挑战，并发布了首个埃及阿拉伯语习语数据集用于进一步研究。
摘要：
我们对大型语言模型（LLMs）处理文化相关语言的能力进行了全面评估，特别是理解并运用隐喻表达的能力，这些表达编码了本地知识和文化细微差别。使用隐喻语言作为文化细微差别和本地知识的代理，我们设计了针对语境理解、语用使用和隐喻解释的评估任务，在阿拉伯语和英语中进行。我们对22个开源和闭源的LLM在埃及阿拉伯语习语、多方言阿拉伯语谚语和英语谚语上进行了评估。我们的结果表明存在一个一致的等级：阿拉伯谚语的平均准确率比英语谚语低4.29%，而埃及习语的准确率比阿拉伯谚语低10.28%。在语用使用任务中，准确率比理解任务下降14.07%，但提供上下文的习语句子可以提高准确率10.66%。模型在隐喻意义方面也存在困难，在100%注释者一致性的习语中，最高只能达到与人类注释者85.58%的一致性。这些发现表明隐喻语言是文化推理的有效诊断：虽然LLMs通常可以解释隐喻意义，但它们在适当使用方面面临挑战。为了支持未来研究，我们发布了Kinayat，第一个为隐喻理解和语用使用评估设计的埃及阿拉伯语习语数据集。
英文摘要：
We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural nuance. Using figurative language as a proxy for cultural nuance and local knowledge, we design evaluation tasks for contextual understanding, pragmatic use, and connotation interpretation in Arabic and English. We evaluate 22 open- and closed-source LLMs on Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs. Our results show a consistent hierarchy: the average accuracy for Arabic proverbs is 4.29% lower than for English proverbs, and performance for Egyptian idioms is 10.28% lower than for Arabic proverbs. For the pragmatic use task, accuracy drops by 14.07% relative to understanding, though providing contextual idiomatic sentences improves accuracy by 10.66%. Models also struggle with connotative meaning, reaching at most 85.58% agreement with human annotators on idioms with 100% inter-annotator agreement. These findings demonstrate that figurative language serves as an effective diagnostic for cultural reasoning: while LLMs can often interpret figurative meaning, they face challenges in using it appropriately. To support future research, we release Kinayat, the first dataset of Egyptian Arabic idioms designed for both figurative understanding and pragmatic use evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
