GeoGalactica:地球科学领域的科学大语言模型
GeoGalactica: A Scientific Large Language Model in Geoscience
AI总结:
本研究通过地球科学文本预训练与指令微调,构建了300亿参数的地球科学领域最大语言模型GeoGalactica,并开源了相关工具与部分模型 checkpoint。
摘要:
大语言模型(LLM)凭借其通用知识以及解决自然语言处理(NLP)领域各类任务的能力取得了巨大成功。凭借其出色的性能,LLM 为跨学科应用带来了希望,即通过人工智能赋能科学(AI for science,AI4S)推动特定领域的科学发现。与此同时,NLP 技术在地球科学研究与实践中的应用广泛且繁杂,涵盖知识抽取、文档分类、问答系统以及知识发现等多个方向。在本研究中,我们迈出了将 LLM 应用于科学领域的第一步,采用了一种相当直接的方法。我们尝试将 LLM 定向适配到地球科学领域,具体方式是使用海量地球科学文本对模型进行进一步预训练,并基于我们自行收集的指令微调数据集对所得模型进行有监督微调(SFT)。这些工作最终产出了拥有 300 亿参数的 GeoGalactica 模型。据我们所知,这是目前地球科学领域规模最大的语言模型。具体而言,GeoGalactica 是基于 Galactica 模型进一步预训练得到的。我们在包含 650 亿 token 的地球科学相关文本语料库上训练 GeoGalactica,该语料库是目前规模最大的地球科学专属文本语料。随后,我们使用 100 万对指令微调数据对模型进行微调,这些数据中的问题需要专业地球科学知识才能解答。在本技术报告中,我们将详细阐述 GeoGalactica 的各个方面,包括数据收集、数据清洗、基座模型选择、预训练、SFT 以及评估。我们开源了数据处理工具,以及预训练前 3/4 阶段的 GeoGalactica 模型检查点。
英文摘要:
Large language models (LLMs) have achieved huge success for their general knowledge and ability to solve a wide spectrum of tasks in natural language processing (NLP). Due to their impressive abilities, LLMs have shed light on potential inter-discipline applications to foster scientific discoveries of a specific domain by using artificial intelligence (AI for science, AI4S). In the meantime, utilizing NLP techniques in geoscience research and practice is wide and convoluted, contributing from knowledge extraction and document classification to question answering and knowledge discovery. In this work, we take the initial step to leverage LLM for science, through a rather straightforward approach. We try to specialize an LLM into geoscience, by further pre-training the model with a vast amount of texts in geoscience, as well as supervised fine-tuning (SFT) the resulting model with our custom collected instruction tuning dataset. These efforts result in a model GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is the largest language model for the geoscience domain. More specifically, GeoGalactica is from further pre-training of Galactica. We train GeoGalactica over a geoscience-related text corpus containing 65 billion tokens, preserving as the largest geoscience-specific text corpus. Then we fine-tune the model with 1 million pairs of instruction-tuning data consisting of questions that demand professional geoscience knowledge to answer. In this technical report, we will illustrate in detail all aspects of GeoGalactica, including data collection, data cleaning, base model selection, pre-training, SFT, and evaluation. We open-source our data curation tools and the checkpoints of GeoGalactica during the first 3/4 of pre-training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。