ClimateGPT：迈向综合气候变化跨学科研究的 AI
ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change
AI总结：
本文提出 ClimateGPT 模型家族，通过领域预训练、指令微调、分层检索增强、级联翻译与多视角问答，实现气候跨学科研究综合，并以更小规模达到接近 Llama-2-70B Chat 的效果。
摘要：
本文介绍了 ClimateGPT，这是一个用于综合气候变化跨学科研究的领域专用大语言模型家族。我们在一个包含 300B token、面向科学的数据集上从头训练了两个 7B 模型。第一个模型在预训练期间纳入了 4.2B 个领域专用 token，第二个模型则在预训练之后适配到气候领域。此外，ClimateGPT-7B、13B 和 70B 在一个包含 4.2B token 的领域专用数据集上由 Llama 2 持续预训练而来。每个模型都在一个高质量、由人工生成的领域专用数据集上进行指令微调，该数据集是与气候科学家密切合作创建的。为减少幻觉数量，我们针对检索增强优化模型，并提出一种分层检索策略。为提高模型对非英语使用者的可及性，我们提议利用级联机器翻译，并证明该方法能够取得与原生多语言模型相当的表现，同时更容易扩展到大量语言。此外，为应对气候变化内在的跨学科特性，我们考虑了不同的研究视角。因此，除总体答案外，该模型还能够生成聚焦不同视角的深入答案。我们提出了一套气候专用的自动基准来评估 LLM。在这些基准上，ClimateGPT-7B 的表现与规模大十倍的 Llama-2-70B Chat 模型相当，同时不会降低其在通用领域基准上的结果。我们的人工评估证实了在基准中观察到的趋势。所有模型均使用可再生能源进行训练和评估，并公开发布。
英文摘要：
This paper introduces ClimateGPT, a model family of domain-specific large language models that synthesize interdisciplinary research on climate change. We trained two 7B models from scratch on a science-oriented dataset of 300B tokens. For the first model, the 4.2B domain-specific tokens were included during pre-training and the second was adapted to the climate domain after pre-training. Additionally, ClimateGPT-7B, 13B and 70B are continuously pre-trained from Llama~2 on a domain-specific dataset of 4.2B tokens. Each model is instruction fine-tuned on a high-quality and human-generated domain-specific dataset that has been created in close cooperation with climate scientists. To reduce the number of hallucinations, we optimize the model for retrieval augmentation and propose a hierarchical retrieval strategy. To increase the accessibility of our model to non-English speakers, we propose to make use of cascaded machine translation and show that this approach can perform comparably to natively multilingual models while being easier to scale to a large number of languages. Further, to address the intrinsic interdisciplinary aspect of climate change we consider different research perspectives. Therefore, the model can produce in-depth answers focusing on different perspectives in addition to an overall answer. We propose a suite of automatic climate-specific benchmarks to evaluate LLMs. On these benchmarks, ClimateGPT-7B performs on par with the ten times larger Llama-2-70B Chat model while not degrading results on general domain benchmarks. Our human evaluation confirms the trends we saw in our benchmarks. All models were trained and evaluated using renewable energy and are released publicly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
