Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at https://github.com/bodasadallah/decrypting-crosswords.
Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models
The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学)
From Commands to Prompts: LLM-based Semantic File System for AIOS
Dalian University of Technology(大连理工大学);Rutgers University(罗格斯大学);Purdue University(普渡大学);New Jersey Institute of Technology(新泽西理工学院);EPFL(洛桑联邦理工学院);University of Minnesota(明尼苏达大学)