The University of Hong Kong(香港大学);University of Cambridge(剑桥大学);City University of Hong Kong(香港城市大学);Peking University(北京大学);Huawei Hisilicon(华为海思);Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港科技大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Autoformalization, the conversion of natural language mathematics into formal languages, offers significant potential for advancing mathematical reasoning. However, existing efforts are limited to formal languages with substantial online corpora and struggle to keep pace with rapidly evolving languages like Lean 4. To bridge this gap, we propose a new benchmark \textbf{Form}alization for \textbf{L}ean~\textbf{4} (\textbf{\name}) designed to evaluate the autoformalization capabilities of large language models (LLMs). This benchmark encompasses a comprehensive assessment of questions, answers, formal statements, and proofs. Additionally, we introduce a \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{PSV}) model that leverages the precise feedback from Lean 4 compilers to enhance autoformalization. Our experiments demonstrate that the PSV method improves autoformalization, enabling higher accuracy using less filtered training data. Furthermore, when fine-tuned with data containing detailed process information, PSV can leverage the data more effectively, leading to more significant improvements in autoformalization for Lean 4. Our dataset and code are available at \url{https://github.com/rookie-joe/PDA}.
LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalization
Department of Statistics, University of Warwick, UK(英国沃里克大学统计系);Center for Advanced Intelligence Project, RIKEN, Japan(日本理化学研究所高级智能项目);Department of Statistics, University of Michigan, USA(美国密歇根大学统计系);Department of Mathematical Informatics, The University of Tokyo(东京大学数学信息学系;日本理化学研究所高级智能项目);also Center for Advanced Intelligence Project, RIKEN, Japan(加州大学伯克利分校电气工程与计算机科学系;统计系);Department of Electrical Engineering and Computer Sciences, also Department of Statistics, University of California, Berkeley, USA(上海交通大学数学科学学院,自然科学院和MOE-LSC);School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China
SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization
Electronics and Telecommunications Research Institute(电子通信研究院);Seoul National University(首尔大学);University of Maryland, College Park(马里兰大学帕克分校);Amazon Web Services(亚马逊网络服务)
FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
The Chinese University of Hong Kong(香港中文大学);Numina;Westlake University(西湖大学);M-A-P;University of California, Los Angeles(加州大学洛杉矶分校);Max Planck Institute for Intelligent Systems, Tübingen(蒂宾根马克斯·普朗克智能系统研究所)
The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements
Department of Computer Science, Informatics Institute, Istanbul Technical University, İstanbul, Türkiye(信息学院计算机科学系,伊斯坦布尔技术大学,伊斯坦布尔,土耳其);Department of Computer Science(计算机科学系);Engineering, Jashore University of Science(工程系,贾沙尔大学科学学院)
MASA: LLM-Driven Multi-Agent Systems for Autoformalization
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);Idiap Research Institute(Idiap研究 institute);National Biomarker Centre, CRUK Manchester Institute(国家生物标志物中心,CRUK曼彻斯特研究所)
FMC: Formalization of Natural Language Mathematical Competition Problems
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China(多媒体信息处理国家重点实验室,计算机科学学院,PKU-Anker LLM实验室,北京大学,北京,中国);Paul G. Allen School of Computer Science(保罗·G·阿伦计算机科学学校);Engineering, University of Washington, U.S.A.(工程,华盛顿大学,美国)
KELPS: A Framework for Verified Multi-Language Autoformalization via Semantic-Syntactic Alignment
School of Information Science and Technology, University of Science and Technology of China(信息科学与技术学院,中国科学技术大学);USTC Knowledge Computing Lab(中国科学技术大学知识计算实验室)
Resource-Lean Lexicon Induction for German Dialects
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(MaiNLP,信息与语言处理中心,慕尼黑大学,德国);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国)
From Keywords to Clusters: AI-Driven Analysis of YouTube Comments to Reveal Election Issue Salience in 2024
Georgia Tech-Europe, Sam Nunn School of International Affairs(佐治亚理工学院欧洲分校萨姆·纳恩国际事务学院);University of Missouri Institute for Data Science and Informatics(密苏里大学数据科学与信息学研究所);University of Missouri School of Medicine(密苏里大学医学院);University of Missouri(密苏里大学)
MechMath: Sorrifier-Driven Formal Decomposition Workflow for Automated Theorem Proving
Academy of Mathematics and Systems Science, CAS(数学与系统科学研究院,中国科学院);School of Advanced Interdisciplinary Sciences, UCAS(交叉科学学院,中国科学院大学);School of Mathematical Science, UCAS(数学科学学院,中国科学院大学)