Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at https://github.com/mathllm/MathCoder2 .
Multilingual Language Model Pretraining using Machine-translated Data
Centre for Artificial Intelligence, University College London(伦敦大学学院人工智能中心);Together AI;Mila, McGill University(米拉-麦吉尔大学);Research and Development Center for Large Language Models, National Institute of Informatics(国立信息学研究所大语言模型研发中心)
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究院);Department of Software Science, Tallinn University of Technology(塔林技术大学软件科学系);Institute of the Estonian Language, Tallinn, Estonia(爱沙尼亚语言研究院);School of Humanities, Tallinn University(塔林大学人文学院)
Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
Institute of Computing (IC) University of Campinas (UNICAMP)(计算研究所(IC)坎皮纳斯大学(UNICAMP));School of Electrical and Computer Engineering (FEEC) University of Campinas (UNICAMP)(电气与计算机工程学院(FEEC)坎皮纳斯大学(UNICAMP))
Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Concordia University(康考迪亚大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Laval University(拉瓦尔大学);University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心)
KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report
AI & ML Group, TU Darmstadt(人工智能与机器学习小组,德累斯顿技术大学);Lab1141;Lamarr Institute(拉马尔研究所);Fraunhofer IAIS(弗劳恩霍夫人工智能研究所);hessian.AI(海斯坦.AI);German Research Center for AI (DFKI)(德国人工智能研究中心(DFKI));Centre for Cognitive Science, TU Darmstadt(认知科学中心,德累斯顿技术大学)
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
McGill University(麦吉尔大学);Mila-Quebec AI Institute(魁北克AI研究所);LMU Munich & Munich Center for Machine Learning(慕尼黑大学及慕尼黑机器学习中心);Microsoft AI for Good Research Lab(微软AI for Good研究实验室)
From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Minzu University of China(民族大学);Institute for AI, Tsinghua University(清华大学人工智能研究院)
Multilingual Multimodal Learning with Machine Translated Text
Wuhan University of Science and Technology(武汉科技大学);University Politehnica of Bucharest(布加勒斯特理工大学);University of Copenhagen(哥本哈根大学);Pioneer Centre for AI(人工智能先锋中心)
Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations
Digital Workforce Services(数字劳动力服务);Heart and Lung Center, Helsinki University Hospital(赫尔辛基大学医院心脏和肺中心);Central Finland Biobank(中央芬兰生物库);Wellbeing Services County of Central Finland(中央芬兰县福祉服务)
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining
Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Bosch Center for Artificial Intelligence(博世人工智能中心)
Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
Department of Pathology, West China Hospital, Sichuan University(四川大学华西医院病理科部门);Institute of Clinical Pathology, West China Hospital, Sichuan University(四川大学华西医院临床病理科研究所);University of Toronto(多伦多大学);Business School, Sichuan University(四川大学商学院);Department of Pathology, Shengjing Hospital of China Medical University(中国医科大学盛京医院病理科部门)