大语言模型是优秀的 cryptic crossword(隐晦填字游戏)解谜者吗?
Are LLMs Good Cryptic Crossword Solvers?
AI总结:
该研究首次测试大语言模型在隐晦填字游戏任务上的能力,针对LLaMA2、Mistral、ChatGPT建立基准,发现三者表现均远逊于人类。
摘要:
隐晦填字游戏(cryptic crosswords)这类谜题不仅依赖常识,还需要解谜者具备在不同层面操控语言、处理各类文字游戏的能力。已有研究表明,即便对现代自然语言处理(NLP)模型而言,解答这类谜题也颇具挑战。但大语言模型(LLMs)在该任务上的能力尚未得到测试。本文针对三款主流LLM——LLaMA2、Mistral和ChatGPT——建立了基准测试结果,显示它们在该任务上的表现仍远不及人类。
英文摘要:
Cryptic crosswords are puzzles that rely not only on general knowledge but also on the solver's ability to manipulate language on different levels and deal with various types of wordplay. Previous research suggests that solving such puzzles is a challenge even for modern NLP models. However, the abilities of large language models (LLMs) have not yet been tested on this task. In this paper, we establish the benchmark results for three popular LLMs -- LLaMA2, Mistral, and ChatGPT -- showing that their performance on this task is still far from that of humans.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。