With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models ($\le 8B$). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.
Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);Graduate College for Engineers, Beijing University of Posts and Telecommunications(北京邮电大学研究生院工程师学院);School of Mathematical Sciences, Fudan University(复旦大学数学科学学院);School of Cyberspace Security, Beijing University of Posts and Telecommunications(北京邮电大学网络空间安全学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);Chu Kochen Honors College, Zhejiang University(浙江大学竺可桢学院);Department of Psychological and Cognitive Sciences, Tsinghua University(清华大学心理学与认知科学系);State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学虚拟现实技术与系统国家重点实验室);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling
School of Artificial Intelligence, Beihang University(北航人工智能学院);Hangzhou International Innovation Institute, Beihang University(北航杭州国际创新研究院);Nanyang Technological University(南洋理工大学);University of Leicester(莱斯特大学);Kuaishou Technology(快手科技)
Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification
School of Computer Science, National Key Laboratory for Multimedia Information Processing, PKU-Anker LLM Lab, Peking University(计算机学院、多媒体信息处理国家重点实验室、PKU-Anker LLM实验室、北京大学);Huawei Noah’s Ark Lab(华为诺亚实验室);The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong Polytechnic University(香港理工大学)
MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics
Department of Computer Science, University of Virginia(弗吉尼亚大学计算机科学系);Astronomy , California Institute of Technology(加州理工学院天文学系);Purdue University(普渡大学);Massachusetts Institute of Technology(麻省理工学院)
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所);Applied AI Institute(应用人工智能研究所);Charles University(查尔斯大学);Nazarbayev University(纳扎尔拜大学);Inria(法国国家信息与自动化研究所);Indian Institute of Technology, Kharagpur (IIT Kharagpur)(印度Kharagpur理工学院);German University of Digital Science(德国数字科学大学)
ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽学校);OpenDataLab, Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)