Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only the final answer but also the intermediate steps is important for evaluation, development, and identifying failures during inference time. To bridge this gap, we introduce ToolComp, a comprehensive benchmark designed to evaluate multi-step tool-use reasoning. ToolComp is developed through a collaboration between models and human annotators, featuring human-edited/verified prompts, final answers, and process supervision labels, allowing for the evaluation of both final outcomes and intermediate reasoning. Evaluation across six different model families demonstrates the challenging nature of our dataset, with the majority of models achieving less than 50% accuracy. Additionally, we generate synthetic training data to compare the performance of outcome-supervised reward models (ORMs) with process-supervised reward models (PRMs) to assess their ability to improve complex tool-use reasoning as evaluated by ToolComp. Our results show that PRMs generalize significantly better than ORMs, achieving a 19% and 11% improvement in rank@1 accuracy for ranking base and fine-tuned model trajectories, respectively. These findings highlight the critical role of process supervision in both the evaluation and training of AI models, paving the way for more robust and capable systems in complex, multi-step tool-use tasks.
Rewarding Novel Deductions: Solver-guided Process Supervision for Logical Reasoning
Information Technology University(信息技术大学);Huazhong Agricultural University(华中农业大学);Qatar Computing Research Institute, Hamad Bin Khalifa University(卡塔尔计算研究所,哈马德·本·哈利法大学)
LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision
National University of Singapore(新加坡国立大学);University College London(伦敦大学学院);University of Manchester(曼彻斯特大学);University of Melbourne(墨尔本大学);University of California, Santa Barbara(加州大学圣巴巴拉分校)
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
University of Chicago(芝加哥大学);Peking University(北京大学);University of California, Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);UNC-Chapel Hill(北卡罗来纳大学教堂山分校)
ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司);Tianjin University(天津大学)
DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA
Department of Information Science, University of Pittsburgh(信息科学系,匹兹堡大学);Department of Biomedical Informatics, University of Pittsburgh(生物医学信息学系,匹兹堡大学);Department of Health Information Management and ISP, University of Pittsburgh(健康信息管理与信息科学系,匹兹堡大学)
Reasoning under Ambiguity: Uncertainty-Aware Multilingual Emotion Classification under Partial Supervision
BUBT Research Graduate School, Bangladesh University of Business and Technology(孟加拉国商业技术大学BUBT研究研究生学院);Department of Computer Science Applied College, University of Ha’il(哈伊大学计算机科学应用学院);Electrical and Computer Engineering, University of Central Florida(佛罗里达中央大学电气与计算机工程学院);Electrical and Computer Engineering, Uttara University(乌塔拉大学电气与计算机工程学院);Department of Computer Science and Engineering, Bangladesh University of Business and Technology(孟加拉国商业技术大学计算机科学与工程系)
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision
University of California, Santa Cruz(加州大学圣克鲁兹分校);Allen Institute for AI(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);University of Maryland, College Park(马里兰大学学院公园分校)
What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis
Ningbo Institute of Digital Twin, Eastern Institute of Technology(宁波数字孪生研究院,东方理工大学);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系)
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦大学玛丽女王学院电子工程与计算机科学学院);The Alan Turing Institute(艾伦·图灵研究所)
Optimizing Language Model's Reasoning Abilities with Weak Supervision
University of California, San Diego(加利福尼亚大学圣迭戈分校);University of Southern California(南加利福尼亚大学);University of Pennsylvania(宾夕法尼亚大学);Yale University(耶鲁大学);Washington University in St. Louis(圣路易斯华盛顿大学)
KnowCoder-A1: Incentivizing Agentic Reasoning Capability with Outcome Supervision for KBQA
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);State Key Laboratory of AI Safety(人工智能安全国家重点实验室);School of Computer Science, University of Chinese Academy of Sciences(中国科学院大学计算机科学学院)
KG-TRACES: Enhancing Large Language Models with Knowledge Graph-constrained Trajectory Reasoning and Attribution Supervision
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学)
Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models
KAIST AI(KAIST人工智能研究所);University of Texas, Austin(德克萨斯大学奥斯汀分校);Rice University(里士满大学);Princeton University(普林斯顿大学);University of Washington(华盛顿大学);Sentient Labs(Sentient实验室)
Learning to Adapt SFT Data for Better Reasoning Generalization
Beihang University(北京航空航天大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);Hangzhou International Innovation Institute(杭州国际创新研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高级创新中心)