连点成线：使用纽约时报Connections文字游戏评估LLMs的抽象推理能力
Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game
AI总结：
本文使用438个《纽约时报》Connections游戏评估LLMs的抽象推理能力，发现表现最好的Claude 3.5 Sonnet仅解决18%的游戏且不及人类玩家，并构建知识类型分类法揭示了LLMs在百科知识等非语义推理上的不足。
摘要：
《纽约时报》Connections游戏已成为文字解谜爱好者流行且具挑战性的消遣。我们收集了438个Connections游戏，以评估最先进的大型语言模型（LLMs）与专家及新手人类玩家的表现对比。结果显示，表现最好的LLM Claude 3.5 Sonnet（尽管它在各种基准上展现了出色的推理能力），也只能完全解决18%的游戏。新手和专家玩家的表现均优于Claude 3.5 Sonnet，其中专家玩家显著超越它。我们创建了成功在Connections游戏中聚类和分类单词所需的知识类型分类法。我们发现，尽管LLMs在基于语义关系对单词进行分类方面表现相对较好，但在其他类型的知识上存在困难，如百科知识、多词表达，或结合单词形式与含义的知识。我们的结果确立了《纽约时报》Connections游戏作为评估AI系统抽象推理能力的挑战性基准。
英文摘要：
The New York Times Connections game has emerged as a popular and challenging pursuit for word puzzle enthusiasts. We collect 438 Connections games to evaluate the performance of state-of-the-art large language models (LLMs) against expert and novice human players. Our results show that even the best performing LLM, Claude 3.5 Sonnet, which has otherwise shown impressive reasoning abilities on a wide variety of benchmarks, can only fully solve 18% of the games. Novice and expert players perform better than Claude 3.5 Sonnet, with expert human players significantly outperforming it. We create a taxonomy of the knowledge types required to successfully cluster and categorize words in the Connections game. We find that while LLMs perform relatively well on categorizing words based on semantic relations they struggle with other types of knowledge such as Encyclopedic Knowledge, Multiword Expressions or knowledge that combines both Word Form and Meaning. Our results establish the New York Times Connections game as a challenging benchmark for evaluating abstract reasoning capabilities in AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
