arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.09012cs.CLcs.AI
收藏

什么让隐晦填字游戏对大语言模型(LLMs)具有挑战性?

What Makes Cryptic Crosswords Challenging for LLMs?

MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Abdelrahman Sadallah, Daria Kotova, Ekaterina Kochmar

AI总结:

该研究针对LLMs在隐晦填字游戏任务上表现差的原因探究不足的问题,测试了Gemma2、LLaMA3、ChatGPT三款主流LLM的基准表现,证实其显著低于人类水平,并探究了性能瓶颈原因,同时开源了相关代码与数据集。

摘要:

隐晦填字游戏是一类依赖常识以及解题者在不同层面操控语言、处理各类文字游戏能力的谜题。已有研究表明,即便对于包括大语言模型(LLMs)在内的现代自然语言处理(NLP)模型,解答这类谜题也颇具难度。但目前极少甚至没有研究探究它们在该任务上表现不佳的原因。本文中,我们针对三款主流LLM——Gemma2、LLaMA3和ChatGPT——建立了基准结果,显示它们在该任务上的表现仍显著低于人类水平。我们还探究了这些模型难以取得更优表现的原因。我们已在https://github.com/bodasadallah/decrypting-crosswords 发布了代码及引入的数据集。

英文摘要:

Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at https://github.com/bodasadallah/decrypting-crosswords.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑