Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a subsequent test-time compute strategy and budget? To explore this, we focus on pass@N, a simple test-time strategy that searches for a correct answer in $N$ independent samples. We show, surprisingly, that training with cross-entropy (CE) loss can be ${\it misaligned}$ with pass@N in that pass@N accuracy ${\it decreases}$ with longer training. We explain the origins of this misalignment in terms of model overconfidence induced by CE, and experimentally verify our prediction of overconfidence as an impediment to scaling test-time compute via pass@N. Furthermore we suggest a principled, modified training loss that is better aligned to pass@N by limiting model confidence and rescuing pass@N test performance. Our algorithm demonstrates improved mathematical reasoning on MATH and MiniF2F benchmarks under several scenarios: (1) providing answers to math questions; and (2) proving theorems by searching over proof trees of varying shapes. Overall our work underscores the importance of co-designing two traditionally separate phases of LLM development: training-time protocols and test-time search and reasoning strategies.
Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Tencent(腾讯);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models
Virginia Tech(弗吉尼亚理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of California, Davis(加州大学戴维斯分校)
GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Alibaba Group(阿里巴巴集团)
ToLeaP: Rethinking Development of Tool Learning with Large Language Models
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Dalian University of Technology(大连理工大学);University of Sydney(悉尼大学);Waseda University(早稻田大学);National University of Singapore(新加坡国立大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
Urban Robotics Lab, School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Republic of Korea(乌尔班机器人实验室,电气工程学院,韩国科学技术院,大韩民国)
CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering
School of Information Technology and Computer Science, Nile University(信息科技与计算机科学学院,尼罗大学);Graduate School of Information Science, University of Hyogo(信息科学研究生院,播磨大学);Advanced Medical Engineering Research Institute, University of Hyogo(先进医疗工程研究所,播磨大学)
Confidence is Not Reliability: Rethinking MC Dropout in Brain Tumour Segmentation
Centre for Doctoral Training in AI for Medical Diagnosis and Care(人工智能辅助医疗诊断与护理博士培训中心);School of Computing, University of Leeds(利兹大学计算机学院);School of Computer Science, University of Leeds(利兹大学计算机科学学院);Leeds Cancer Centre, St James’s University Hospital, Leeds, UK(利兹癌症中心,圣詹姆斯大学医院,利兹,英国)
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Shanghai Jiao Tong University(上海交通大学);GAIR;Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Institute(上海创新研究院)
Rethinking the Potential of Layer Freezing for Efficient DNN Training
University of Georgia(佐治亚大学);Northeastern University(东北大学);University of Pittsburgh(匹兹堡大学);The University of Arizona(亚利桑那大学);Stevens Institute of Technology(斯蒂文斯理工学院);The University of North Carolina at Charlotte(北卡罗来纳大学教堂山分校);University of Virginia(弗吉尼亚大学)