OpenAI's o3 achieves a high score of 87.5 % on ARC-AGI, a benchmark proposed to measure intelligence. This raises the question whether systems based on Large Language Models (LLMs), particularly o3, demonstrate intelligence and progress towards artificial general intelligence (AGI). Building on the distinction between skills and intelligence made by François Chollet, the creator of ARC-AGI, a new understanding of intelligence is introduced: an agent is the more intelligent, the more efficiently it can achieve the more diverse goals in the more diverse worlds with the less knowledge. An analysis of the ARC-AGI benchmark shows that its tasks represent a very specific type of problem that can be solved by massive trialling of combinations of predefined operations. This method is also applied by o3, achieving its high score through the extensive use of computing power. However, for most problems in the physical world and in the human domain, solutions cannot be tested in advance and predefined operations are not available. Consequently, massive trialling of predefined operations, as o3 does, cannot be a basis for AGI - instead, new approaches are required that can reliably solve a wide variety of problems without existing skills. To support this development, a new benchmark for intelligence is outlined that covers a much higher diversity of unknown tasks to be solved, thus enabling a comprehensive assessment of intelligence and of progress towards AGI.
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
Technology Innovation Institute（技术创新研究院）；University of Oslo（奥斯陆大学）；Eötvös Loránd University（厄特沃什·罗兰大学）；University of Guelma（盖勒马大学）；The University of Manchester（曼彻斯特大学）；Khalifa University（哈利法大学）；Cyentific AS（赛恩蒂菲克公司）
Benchmarking Large Language Models with Integer Sequence Generation Tasks
Earth and Environmental Sciences Division Los Alamos National Laboratory（地球与环境科学 division 洛斯阿拉莫斯国家实验室）；Theoretical Division Los Alamos National Laboratory（理论 division 洛斯阿拉莫斯国家实验室）；Computational Sciences Division Los Alamos National Laboratory（计算科学 division 洛斯阿拉莫斯国家实验室）
Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification
Department of Information Science and Technology（信息科学与技术系）；College of Emergency Preparedness, Homeland Security, and Cybersecurity（应急准备、国土安全与网络安全学院）；University at Albany, SUNY（萨利纳大学）
Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology
Centre for Responsible Autonomous Systems in Healthcare (CRASH) Lab, Koita Centre for Digital Health（负责任的自主医疗系统中心（CRASH）实验室、Koita数字健康中心）；Ashoka University（阿什oka大学）；Independent Researcher（独立研究者）；Koita Centre for Digital Health（Koita数字健康中心）
VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models
Zhejiang University（浙江大学）；Meituan Group（美团集团）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
Benchmarking large language models for materials synthesis: the case of atomic layer deposition
Applied Materials Division, Argonne National Laboratory（阿贡国家实验室应用材料部门）；Business and Information Systems, Argonne National Laboratory（阿贡国家实验室商业与信息系统部门）；Northwestern Center for Water Research, Northwestern University（西北大学水研究中心）
BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
National University of Singapore（新加坡国立大学）；Yale University（耶鲁大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡全国眼科中心）；Massachusetts Eye and Ear（麻省眼耳医院）；Harvard Medical School（哈佛医学院）；Nanyang Technological University（南洋理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
National University of Singapore（新加坡国立大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡国家眼科中心）；The Chinese University of Hong Kong（香港中文大学）；Yale University（耶鲁大学）；Cleveland Clinic（克利夫兰诊所）；Montreal（蒙特利尔）；Tan Tock Seng Hospital（陈笃生医院）；Stanford University（斯坦福大学）
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
ByteDance Seed（字节跳动种子）；Fudan University（复旦大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；SIA-Lab of Tsinghua AIR and ByteDance Seed（清华大学AIR实验室和字节跳动种子）
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University（四川大学计算机学院）；School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）；Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University（中国电信与西北工业大学人工智能研究院（TeleAI））
