Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning
Shanghai AI Lab(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);The University of Hong Kong(香港大学);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models
University College London(伦敦大学学院);University of Liverpool(利物浦大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Westlake University(西湖大学)
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign(计算机科学系伊利诺伊大学厄巴纳-香槟分校);Department of Computer Science Stanford University(计算机科学系斯坦福大学)
MMATH: A Multilingual Benchmark for Mathematical Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);iFLYTEK Research (Central China), iFLYTEK Co., Ltd.(iFLYTEK中央研究院、iFLYTEK有限公司)
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
Shanghai AI Laboratory(上海人工智能实验室);MMLab, The Chinese University of Hong Kong(香港中文大学多媒体实验室);Shanghai Jiao Tong University(上海交通大学);Great Bay University(大湾区大学)
A Survey of Task-Oriented Knowledge Graph Reasoning: Status, Applications, and Prospects
School of Artificial Intelligence, Beihang University(北航人工智能学院);School of Information Science and Technology, Beijing Forestry University(北京林业大学信息科学与技术学院)
NLP(自然语言处理);CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系计算机视觉实验室,澳门大学);University of Macau(澳门大学);DAMO Academy, Alibaba Group(阿里集团达摩院)
Georgia Institute of Technology(佐治亚理工学院);Seoul National University(首尔国立大学);Microsoft(微软);Massachusetts Institute of Technology(麻省理工学院);Brown University(布朗大学)
A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);Fudan University(复旦大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hong Kong Polytechnic University(香港理工大学);University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);Stanford University(斯坦福大学);University of California, Merced(加州大学默塞德分校)
CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Idiap Research Institute(Idiap研究机构);National Biomarker Centre, CRUK-MI, Univ. of Manchester(国家生物标记中心,CRUK-MI,曼彻斯特大学)
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
Shanghai Jiao Tong University(上海交通大学);SII;SPIRAL Lab(SPIRAL实验室);Generative AI Research Lab (GAIR)(生成式AI研究实验室);Shanghai Chest Hospital(上海胸科医院);Beijing Anzhen Hospital, Capital Medical University(北京安贞医院,首都医科大学)