Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 613 problems based on the NPR Sunday Puzzle Challenge that requires only general knowledge. Our benchmark is challenging for both humans and models; however correct solutions are easy to verify, and models' mistakes are easy to spot. As LLMs are more widely deployed in society, we believe it is useful to develop benchmarks for frontier models that humans can understand without the need for deep domain expertise. Our work reveals capability gaps that are not evident in existing benchmarks: OpenAI o1 significantly outperforms other reasoning models on our benchmark, despite being on par with other models when tested on benchmarks that test specialized knowledge. Furthermore, our analysis of reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance, often concedes with "I give up" before providing an answer that it knows is wrong. R1 can also be remarkably "uncertain" in its output and in rare cases, it does not "finish thinking," which suggests the need for techniques to ``wrap up'' before the context window limit is reached. We also quantify the effectiveness of reasoning longer to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.
Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning
Department of Electrical and Computer Engineering, Seoul National University（首尔国立大学电子与计算机工程系）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔国立大学人工智能跨学科项目）；AIIS, ASRI, INMC, and ISRC, Seoul National University（首尔国立大学人工智能研究所、人工智能研究室、智能网络研究中心和智能系统研究中心）；DGIST（延世大学）
Reasoning on Time-Series for Financial Technical Analysis
National University of Singapore（国立新加坡大学）；Technical University of Munich（慕尼黑技术大学）；Singapore Management University（新加坡管理学院）；City University of Hong Kong（香港城市大学）
The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge. Although prior work has sought to increase the knowledge coverage by incorporating structured knowledge beyond text, accessing heterogeneous knowledge sources through a unified interface remains an open question. While data-to-text generation has the potential to serve as a universal interface for data and text, its feasibility for downstream tasks remains largely unknown. In this work, we bridge this gap and use the data-to-text method as a means for encoding struc…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）
We administered the Verbal IQ (VIQ) part of the Wechsler Preschool and Primary Scale of Intelligence (WPPSI-III) to the ConceptNet 4 AI system. The test questions (e.g., "Why do we shake hands?") were translated into ConceptNet 4 inputs using a combination of the simple natural language processing tools that come with ConceptNet together with short Python programs that we wrote. The question answering used a version of ConceptNet based on spectral methods. The ConceptNet system scored a WPPSI-III VIQ that is average for a four-year-old child, but below average for 5 to 7 year-olds. Large varia…
Revisiting Uncertainty Estimation and Calibration of Large Language Models
School of Computer Science University of Sydney（悉尼大学计算机科学学院）；City University of Hong Kong（香港城市大学）；Shanghai Jiao Tong University（上海交通大学）；Department of Engineering Science University of Oxford（牛津大学工程科学系）
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Tsinghua University（清华大学）；Behavioral and Spatial AI Lab, Peking University & Tongji University（北京大学与同济大学行为与空间人工智能实验室）
Theoretically as well as experimentally it is investigated how people represent their knowledge in order to make decisions or to share their knowledge with others. Experiment 1 probes into the ways how people 6ather information about the frequencies of events and how the requested response mode, that is, numerical vs. verbal estimates interferes with this knowledge. The least interference occurs if the subjects are allowed to give verbal responses. From this it is concluded that processing knowledge about uncertainty categorically, that is, by means of verbal expressions, imposes less mental w…
Comparing verbal, visual and combined explanations for Bayesian Network inferences
Dept of Data Science and AI, Faculty of Information Technology Monash University, Australia（数据科学与人工智能系，信息科技学院，墨尔本大学，澳大利亚）；Dept of Human-Centred Computing, Faculty of Information Technology Monash University, Australia（以人为本计算系，信息科技学院，墨尔本大学，澳大利亚）；Faculty of Information Technology Monash University, Australia（信息科技学院，墨尔本大学，澳大利亚）
This article describes a novel approach to expand in run-time the knowledge base of an Artificial Conversational Agent. A technique for automatic knowledge extraction from the user's sentence and four methods to insert the new acquired concepts in the knowledge base have been developed and integrated into a system that has already been tested for knowledge-based conversation between a social humanoid robot and residents of care homes. The run-time addition of new knowledge allows overcoming some limitations that affect most robots and chatbots: the incapability of engaging the user for a long …
