Large language models (LLMs) have shown excellent mastering of human language, but still struggle in real-world applications that require mathematical problem-solving. While many strategies and datasets to enhance LLMs' mathematics are developed, it remains a challenge to simultaneously maintain and improve both language and mathematical capabilities in deployed LLM systems.In this work, we tailor the Self-Critique pipeline, which addresses the challenge in the feedback learning stage of LLM alignment. We first train a general Math-Critique model from the LLM itself to provide feedback signals. Then, we sequentially employ rejective fine-tuning and direct preference optimization over the LLM's own generations for data collection. Based on ChatGLM3-32B, we conduct a series of experiments on both academic and our newly created challenging dataset, MathUserEval. Results show that our pipeline significantly enhances the LLM's mathematical problem-solving while still improving its language ability, outperforming LLMs that could be two times larger. Related techniques have been deployed to ChatGLM\footnote{\url{https://chatglm.cn}}, an online serving LLM. Related evaluation dataset and scripts are released at \url{https://github.com/THUDM/ChatGLM-Math}.
Training Language Model to Critique for Better Refinement
China Telecom Research Institute(中国电信研究院);The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);University of Electronic Science and Technology of China(电子科技大学);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组);Zhipu AI(智谱AI)
DeepCritic: Deliberate Critique with Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学)
Self-Evolving Critique Abilities in Large Language Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Shenzhen International Center for Industrial and Applied Mathematics(深圳国际工业与应用数学中心);Qwen Team, Alibaba Inc.(阿里云通义团队)
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc(小红书公司)
ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Zhejiang University(浙江大学);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理重点实验室)
Position: LLMs Must Use Functor-Based and RAG-Driven Bias Mitigation for Fairness
Knight Foundation School of Computing and Information Sciences(骑士基金会计算与信息科学学院);Florida International University(佛罗里达国际大学);College of Engineering(工程学院);University of South Florida(佛罗里达州立大学)
CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、BNRist、清华大学);School of Electronic and Information Engineering, Beijing Jiaotong University(电子信息工程学院、北京交通大学);Department of Electronic Engineering, Tsinghua University(电子工程系、清华大学);Department of Urban Planning, Tsinghua University(城市规划系、清华大学)
Advancing LLM Reasoning Generalists with Preference Trees
Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Northeastern University(东北大学);Renmin University of China(中国人民大学);BUPT(北京邮电大学);Tencent(腾讯)