评估大型语言模型对乌尔都语习语的处理能力
Evaluating Large Language Models on Urdu Idioms
AI总结：
本研究构建含4000组双语习语对的乌尔都语跨文字习语评估基准，采用多种提示策略评估前沿LLMs，发现其在习语处理上优于传统NMT，凸显提示工程的重要性。
摘要：
习语因具有比喻性和文化根基的含义，与字面表达存在区别，仍是自然语言处理领域长期存在的挑战。尽管近期大型语言模型（LLMs）的进展提升了多种语言的习语处理能力，但乌尔都语等低资源语言却未受到足够关注。本研究构建了一套乌尔都语-英语习语翻译综合基准，包含经人工验证的4000组对齐习语句子对，涵盖波斯-阿拉伯文（乌尔都语原生文字）和罗马化乌尔都语两种形式。我们采用文化提示、习语提示、少样本学习等多种提示策略，对翻译、 paraphrasing（释义）、习语跨度检测、回译等多项任务进行评估。研究结果显示，前沿大型语言模型在所有评估场景中均持续优于传统神经机器翻译系统，尤其在保留比喻和隐喻含义方面表现突出。模型在乌尔都语原生文字上的性能相对稳定，但罗马化乌尔都语缺乏标准正字法，为一致性及习语跨度检测带来了重大挑战。本研究建立了一套高质量的乌尔都语跨文字习语评估基准，强调了提示工程在跨语言保留比喻性语言含义方面的重要性。
英文摘要：
Idioms remain a persistent challenge in natural language processing due to their figurative and culturally grounded meanings, which distinguish them from literal expressions. Although recent advances in large language models (LLMs) have improved idiom handling across several languages, limited attention has been given to low resource languages such as Urdu. In this work, we present a comprehensive benchmark for Urdu to English idiomatic translation, consisting of a manually verified dataset of 4,000 aligned idiom sentence pairs in both Perso Arabic (native Urdu script) and Romanized Urdu. We evaluate multiple tasks, including translation, paraphrasing, idiom span detection, and back-translation, using diverse prompting strategies such as cultural prompting, idiomatic prompting, and few-shot learning. Our findings show that frontier LLMs consistently outperform traditional neural machine translation systems across all evaluation settings, particularly in preserving figurative and metaphorical meaning. While models demonstrate relatively stable performance on native Urdu script, the absence of standardized orthography in Romanized Urdu introduces substantial challenges for consistency and idiom span detection. This work establishes a high quality benchmark for cross script idiomatic evaluation in Urdu and underscores the importance of prompt engineering in preserving figurative language meaning across languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
