The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past years. In response, we present JEEBench, a considerably more challenging benchmark dataset for evaluating the problem solving abilities of LLMs. We curate 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deep in-domain knowledge is essential for solving problems in this benchmark. Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40%. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. We also observe that by mere prompting, GPT-4 is unable to assess risk introduced by negative marking for incorrect answers. For this, we develop a post-hoc confidence-thresholding method over self-consistency, which enables effective response selection. We hope that our challenging benchmark will guide future re-search in problem-solving using LLMs.
Puzzle Solving using Reasoning of Large Language Models: A Survey
Artificial Intelligence and Learning Systems Laboratory(人工智能与学习系统实验室);School of Electrical and Computer Engineering(电气与计算机工程学院);National Technical University of Athens(希腊雅典国家技术大学)
NLPBench: Evaluating Large Language Models on Solving NLP Problems
Waseda University(早稻田大学);University of Tokyo(东京大学);University of Washington(华盛顿大学);Zhejiang Lab(之江实验室);University of Science and Technology of China(中国科学技术大学);University of Maryland(马里兰大学)
One More Question is Enough, Expert Question Decomposition (EQD) Model for Domain Quantitative Reasoning
The University of Edinburgh(爱丁堡大学);National Technical University of Athens(雅典技术大学);Archimedes/Athena Research Centre(阿基米德/雅典研究中心);University of Aveiro(阿维罗大学)
Developing an LLM-Based Feedback System Grounded in Evidence-Centered Design to Support Physics Problem Solving
Leibniz Institute for Science and Mathematics Education(莱布尼茨科学和数学教育研究所);Free University of Berlin(柏林自由大学);Ludwigsburg University of Education(路德维希堡教育大学)