arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2312.03360cs.CLcs.AIcs.LG
收藏

通过额外训练将特定科学知识注入大语言模型

Teaching Specific Scientific Knowledge into Large Language Models through Additional Training

Tokyo Institute of Technology（东京工业大学）；University of Tsukuba（筑波大学）；The University of Tokyo（东京大学）

Kan Hatakeyama-Sato, Yasuhiko Igarashi, Shun Katakami, Yuta Nabae, Teruaki Hayakawa

备注：
added token information for some texts, and fixed typo
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究探索通过额外训练向Llama 2注入科学知识，采用文本增强、超参数优化等方法，用6.5万篇论文验证，实现部分知识嵌入并指明后续改进方向。

摘要：

我们通过额外训练，探索将专业科学知识嵌入Llama 2大语言模型（LLM）的方法。关键发现表明，有效的知识整合需要从多视角阅读文本，尤其是指令格式的文本。我们利用文本增强技术（包括风格转换和翻译）解决专业文本稀缺的问题。超参数优化至关重要，不同规模的模型（7b、13b和70b）都可合理进行额外训练。为验证方法，我们构建了包含65000篇科学论文的数据集。尽管我们已成功实现部分知识嵌入，但该研究凸显了将专业信息整合进LLM的复杂性与局限性，指明了后续改进方向。

英文摘要：

Through additional training, we explore embedding specialized scientific knowledge into the Llama 2 Large Language Model (LLM). Key findings reveal that effective knowledge integration requires reading texts from multiple perspectives, especially in instructional formats. We utilize text augmentation to tackle the scarcity of specialized texts, including style conversions and translations. Hyperparameter optimization proves crucial, with different size models (7b, 13b, and 70b) reasonably undergoing additional training. Validating our methods, we construct a dataset of 65,000 scientific papers. Although we have succeeded in partially embedding knowledge, the study highlights the complexities and limitations of incorporating specialized information into LLMs, suggesting areas for further improvement.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑