HoneyBee：面向材料科学的大型语言模型的渐进式指令微调
HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science
AI总结：
提出MatSci-Instruct流程构建可信材料科学指令数据，微调LLaMa模型HoneyBee，在MatSci-NLP基准上超越现有模型，实现渐进式性能提升。
摘要：
我们提出了一种基于指令的材料科学可信数据整理流程（MatSci-Instruct），并将其应用于微调一个面向材料科学的基于LLaMa的语言模型（HoneyBee）。MatSci-Instruct有助于缓解公开文献中相关、高质量材料科学文本数据的稀缺性，而HoneyBee是第一个专门针对材料科学的十亿参数语言模型。在MatSci-Instruct中，我们通过提示多个商用大型语言模型进行生成，并引入一个指导模块（如Chat-GPT）以及一个独立的验证模块（如Claude）进行验证，从而提高了生成数据的可信度。利用MatSci-Instruct，我们构建了一个包含多种任务的数据集，并从多个维度衡量数据集的质量，包括与已知事实的准确性、与材料科学的相关性，以及数据的完整性和合理性。此外，我们在微调-评估-反馈循环中迭代生成更有针对性的指令和指令数据，从而使我们微调的HoneyBee模型性能逐步提升。我们在MatSci-NLP基准上的评估显示，HoneyBee在材料科学任务上优于现有语言模型，并在指令数据精炼的连续阶段中实现迭代改进。我们通过自动评估研究HoneyBee语言建模的质量，并分析案例研究以进一步理解模型的能力和局限性。我们的代码和相关数据集可在\url{https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee}公开获取。
英文摘要：
We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available at \url{https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
