Puzzle Solving using Reasoning of Large Language Models: A Survey
Artificial Intelligence and Learning Systems Laboratory（人工智能与学习系统实验室）；School of Electrical and Computer Engineering（电气与计算机工程学院）；National Technical University of Athens（希腊雅典国家技术大学）
Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou
Exploring the capabilities of Large Language Models (LLMs) in puzzle solving unveils critical insights into their potential and challenges in AI, marking a significant step towards understanding their applicability in complex reasoning tasks. This survey leverages a unique taxonomy -- dividing puzzles into rule-based and rule-less categories -- to critically assess LLMs through various methodologies, including prompting techniques, neuro-symbolic approaches, and fine-tuning. Through a critical review of relevant datasets and benchmarks, we assess LLMs' performance, identifying significant challenges in complex puzzle scenarios. Our findings highlight the disparity between LLM capabilities and human-like reasoning, particularly in those requiring advanced logical inference. The survey underscores the necessity for novel strategies and richer datasets to advance LLMs' puzzle-solving proficiency and contribute to AI's logical reasoning and creative problem-solving advancements.
Solving crossword puzzles requires diverse reasoning capabilities, access to a vast amount of knowledge about language and the world, and the ability to satisfy the constraints imposed by the structure of the puzzle. In this work, we introduce solving crossword puzzles as a new natural language understanding task. We release the specification of a corpus of crossword puzzles collected from the New York Times daily crossword spanning 25 years and comprised of a total of around nine thousand puzzles. These puzzles include a diverse set of clues: historic, factual, word meaning, synonyms/antonyms…
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
ByteDance Seed（字节跳动种子）；Fudan University（复旦大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；SIA-Lab of Tsinghua AIR and ByteDance Seed（清华大学AIR实验室和字节跳动种子）
Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Department of Linguistics, Harvard University（哈佛大学语言学系）；Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学凯普纳自然与人工智能研究 institute）
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
Alibaba Group（阿里巴巴集团）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海市智能信息处理重点实验室）
UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed人工智能大学）；IIT Gandhinagar（印度加尔各答理工学院）；Harvard University（哈佛大学）；VinUniversity（文大学）；Universitas Indonesia（印度尼西亚大学）
JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
University of Southern California（南加州大学）；National University of Singapore（新加坡国立大学）；Adobe Research（奥多比研究院）；Texas A&M University（德克萨斯农工大学）；Rice University（莱斯大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
AIML Lab, TU Darmstadt（AIML实验室，德累斯顿技术大学）；Hessian Center for AI (hessian.ai)（黑森人工智能中心）；Institute of Psychology, TU Darmstadt（心理学研究所，德累斯顿技术大学）；Centre for Cognitive Science, TU Darmstadt（认知科学中心，德累斯顿技术大学）；German Center for AI (DFKI)（德国人工智能中心）
