Large Language Models (LLMs) have recently achieved impressive performance in math and reasoning benchmarks. However, they often struggle with logic problems and puzzles that are relatively easy for humans. To further investigate this, we introduce a new benchmark, SearchBench, which contains 11 unique search problems inspired by intuitive puzzles. Each SearchBench problem type is equipped with automated pipelines to generate an arbitrary number of instances and analyze the feasibility, correctness, and optimality of LLM-generated solutions. We show that using step-by-step, language-only reasoning, even the most advanced LLMs fail to solve SearchBench; for example, OpenAI's frontier models GPT-4 and o1-preview solve only 1.4% and 18.6% of problems, respectively. The reason is that SearchBench problems require considering multiple pathways and performing backtracking, posing a significant challenge to auto-regressive models. Interestingly, performance is significantly boosted when we prompt models to generate a complete A* search algorithm - a comparatively more cognitively difficult task. This approach effectively offloads the iterative search and backtracking process from the models, which they struggle with in text. This in-context learning baseline is further enhanced via a Multi-Stage-Multi-Try (MSMT) inference method, increasing GPT-4's rate of correct solutions to over 57%.
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (CAS)(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Wenlan School of Business, Zhongnan University of Economics and Law(中南财经政法大学文澜商学院)
Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM Agent
School of Information Science;Electronic Engineering,\ Jiao Tong University Shanghai China;School of Artificial Intelligence,\ Jiao Tong University Shanghai China;State Key Laboratory of Estuarine;Coastal Research,\ China Normal University Shanghai China;School of Oceanography,\ Jiao Tong University Shanghai China;Institute of Geographical Science;Natural Resources Research,\ Academy of Sciences Beijing China;Electronic Engineering,\ Jiao Tong University;School of Artificial Intelligence,\ Jiao Tong University;Coastal Research,\ China Normal University;School of Oceanography,\ Jiao Tong University;Natural Resources Research,\ Academy of Sciences
Cognition on Graph: Navigating Massive Knowledge Space via Cognitive Cycles and Bidirectional Graph-Text Synergy
Beijing University of Posts and Telecommunications(北京邮电大学);Zhongguancun Academy(中关村学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统全国重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Deep reflective reasoning in interdependence constrained structured data extraction from clinical notes for digital health
Quantitative Biomedical Research Center, Department of Health Data Science and Biostatistics, Peter O'Donnell School of Public Health, University of Texas Southwestern Medical Center(定量生物医学研究中心,健康数据科学与生物统计学系,彼得·奥·多尼尔公共卫生学院,德克萨斯西南医学中心)
From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task
Faculty of Exact and Natural Sciences, University of Buenos Aires(布宜诺斯艾利斯大学精确与自然科学学院);National Scientific and Technical Research Council (CONICET)(国家科学技术研究委员会(CONICET))
Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning
Department of Neurology and Neurologic Sciences(神经病学与神经科学系);Stanford University(斯坦福大学);Image Guided Therapy and Research Facility(影像引导治疗与研究设施);University of Dundee(邓迪大学);Weill Institute of Neurology and Neurosciences(韦尔神经病学与神经科学研究所);University of California, San Francisco(加州大学旧金山分校);Department of Systems and Enterprises(系统与企业系);Stevens Institute of Technology(史蒂文斯理工学院);Department of EECS(电子工程与计算机科学系);University of California Berkeley(加州大学伯克利分校)
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
School of Law, China University of Political Science and Law(中国政法大学法学院);Zhejiang University of Finance and Economics Dongfang College(浙江财经大学东洋学院);School of Information Management for Law, China University of Political Science and Law(中国政法大学信息管理法学院);Department of Artificial Intelligence, Chung-Ang University(成均馆大学人工智能系);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系)
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
Beijing University of Technology(北京工业大学);Beijing Institute of Technology(北京理工大学);China-Japan Friendship Hospital(中日友好医院);Dongbei University of Finance and Economics(东北财经大学)
VCN-Bench: A Video-Contextualized Navigation Benchmark for Spatial Reasoning over Prior Visual Experience
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);Zhejiang University(浙江大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)