FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Epoch AI;University of Leicester(利兹大学);RWTH Aachen University(亚琛工业大学);King’s College London(伦敦大学国王学院);University of Bristol(布里斯托大学);Iowa State University(爱荷华州立大学);MIT(麻省理工学院);University of North Carolina(北卡罗来纳大学);CNRS - Université Paris-Saclay(法国国家科学研究中心-巴黎-萨克雷大学);University of Siegen(锡根大学);Knox College at Charlotte(夏洛特克恩学院);James Madison University(詹姆斯麦迪逊大学);ICMC, USP(巴西圣保罗大学ICMC分校);Masaryk University(马萨里克大学);UC Berkeley(伯克利加州大学);Cornell University(康奈尔大学);Rutgers University(罗格斯大学);Harvard University(哈佛大学);ETH Zurich(苏黎世联邦理工学院);Independent(独立研究者)
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira… 展开作者
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, Mark Wildon
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.
Evaluating Cognitive Age Alignment in Interactive AI Agents
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shenzhen Children's Hospital(深圳儿童医院);Peking University(北京大学);Hong Kong Polytechnic University(香港理工大学)
MAPWise: Evaluating Vision-Language Models for Advanced Map Queries
IIIT Hyderabad(印度信息技术学院海得拉巴分校);University of Utah(犹他大学);University of Delaware(特拉华大学);Arizona State University(亚利桑那州立大学);University of Pennsylvania(宾夕法尼亚大学)
EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能国家重点实验室);University of Hong Kong(香港大学)
EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
InFoBench: Evaluating Instruction Following Ability in Large Language Models
Tencent AI Lab, Seattle(腾讯人工智能实验室(西雅图));University of Central Florida(中佛罗里达大学);University of Georgia(佐治亚大学);Emory University(埃默里大学);Shanghai Jiao Tong University(上海交通大学)
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University(复旦大学);NJU-LINK Team, Nanjing University(南京大学NJU-LINK团队);University of Science and Technology of China(中国科学技术大学);Kuaishou Technology(快手科技)
Fréchet Power-Scenario Distance: A Metric for Evaluating Generative AI Models across Multiple Time-Scales in Smart Grids
Department of Electrical and Computer Engineering, Texas A&M University(电气与计算机工程系,德克萨斯农工大学);John A. Paulson School of Engineering and Applied Sciences, Harvard University(约翰·A·保罗森工程与应用科学学院,哈佛大学)
University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);Emory University(埃默里大学);University of Minnesota(明尼苏达大学);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校);UMass Chan Medical School(UMass Chan医学学院)
Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
Pennsylvania State University(宾夕法尼亚州立大学);City University of New york(纽约城市大学);New York University(纽约大学);Amirkabir University of Technology(阿米尔卡比尔理工大学);Bocconi University(博科尼大学);Autodesk(欧特克公司)