NYT-Connections：一个欺骗性简单的文本分类任务，难倒系统1思考者
NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers
AI总结：
本文提出NYT-Connections基准，通过358个单词分类谜题评估LLM推理，发现即使GPT-4也比人类低近30%，且高级提示技术收益递减，为测试系统1思维缺陷提供新工具。
摘要：
大型语言模型（LLMs）在各种基准测试中表现出色，但其进行深思熟虑推理的能力仍存疑。我们提出了NYT-Connections，一个包含358个简单单词分类谜题的集合，源自《纽约时报》的Connections游戏。该基准旨在惩罚快速、直觉性的“系统1”思维，从而隔离出基础推理能力。我们评估了六个最新的LLM、一个简单的机器学习启发式方法以及人类，在三种配置下进行测试：单次尝试、无提示的多次尝试、以及带上下文提示的多次尝试。我们的研究结果揭示了显著的性能差距：即使是表现最好的LLM（如GPT-4）也比人类表现低近30%。值得注意的是，先进的提示技术（如思维链和自一致性）在任务难度增加时显示出收益递减。NYT-Connections独特地结合了语言隔离、对直觉捷径的抵抗力以及定期更新以减轻数据泄漏，为评估LLM推理能力提供了一种新颖工具。
英文摘要：
Large Language Models (LLMs) have shown impressive performance on various benchmarks, yet their ability to engage in deliberate reasoning remains questionable. We present NYT-Connections, a collection of 358 simple word classification puzzles derived from the New York Times Connections game. This benchmark is designed to penalize quick, intuitive "System 1" thinking, isolating fundamental reasoning skills. We evaluated six recent LLMs, a simple machine learning heuristic, and humans across three configurations: single-attempt, multiple attempts without hints, and multiple attempts with contextual hints. Our findings reveal a significant performance gap: even top-performing LLMs like GPT-4 fall short of human performance by nearly 30%. Notably, advanced prompting techniques such as Chain-of-Thought and Self-Consistency show diminishing returns as task difficulty increases. NYT-Connections uniquely combines linguistic isolation, resistance to intuitive shortcuts, and regular updates to mitigate data leakage, offering a novel tool for assessing LLM reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
