arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.16461cs.CL
收藏

大型语言模型中多语言习语与明喻的对比研究

Comparative Study of Multilingual Idioms and Similes in Large Language Models

Tehran Institute for Advanced Studies, Khatam University(哈塔姆大学德黑兰高等研究院)

Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, Yadollah Yaghoobzadeh

AI总结:

本文通过多语言明喻与习语数据集评估LLMs的比喻语言理解能力,比较思维链、少样本和翻译提示等策略,发现提示工程效果因语言、比喻类型和模型而异,开源模型在低资源语言明喻上表现较差,且习语任务已接近饱和。

摘要:

本研究填补了文献中关于大型语言模型(LLMs)在跨多种语言解释不同类型比喻性语言方面比较性能的空白。通过使用两个关于明喻和习语解释的多语言数据集对LLMs进行评估,我们探索了各种提示工程策略的有效性,包括思维链、少样本和英文翻译提示。我们还通过构建两个新的评估集,将这些数据集的语言扩展到波斯语。我们的综合评估涉及闭源模型(GPT-3.5、GPT-4o mini、Gemini 1.5)和开源模型(Llama 3.1、Qwen2),揭示了不同语言和比喻类型之间性能的显著差异。我们的研究结果表明,虽然提示工程方法总体上是有效的,但其成功程度因比喻类型、语言和模型而异。我们还观察到,开源模型在明喻任务中尤其难以处理低资源语言。此外,许多语言的习语解释已接近饱和,需要更具挑战性的评估。

英文摘要:

This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑