预训练生成语言模型作为基于序列任务的通用学习框架
Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks
AI总结：
本研究提出利用小型预训练生成语言模型作为通用学习框架，通过指令微调在化学信息学任务上达到接近最先进性能，克服了从零训练的资源挑战。
摘要：
我们提出，具有数百万参数的小型预训练基础生成语言模型可以被用作基于序列任务的通用学习框架。我们的提议克服了从零开始训练神经网络和语言模型所面临的计算资源、技能集和时间线挑战。此外，我们的方法专注于创建小型且高度专业化的模型，这些模型能够准确执行基础模型无法完成的挑战性任务。我们证明了具有125M、350M和1.3B参数的预训练基础语言模型可以通过10,000到1,000,000个指令示例进行指令微调，从而在具有挑战性的化学信息学任务上达到接近最先进的结果。我们还展示了连续语言模型微调轮次对改进结果的作用，以及数据格式化和预训练基础语言模型选择对指令微调成功的重要性。
英文摘要：
We propose that small pretrained foundational generative language models with millions of parameters can be utilized as a general learning framework for sequence-based tasks. Our proposal overcomes the computational resource, skill set, and timeline challenges associated with training neural networks and language models from scratch. Further, our approach focuses on creating small and highly specialized models that can accurately execute a challenging task of which the base model is incapable of performing. We demonstrate that 125M, 350M, and 1.3B parameter pretrained foundational language models can be instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve near state-of-the-art results on challenging cheminformatics tasks. We also demonstrate the role of successive language model fine-tuning epochs on improved outcomes, as well as the importance of both data formatting and pretrained foundational language model selection for instruction fine-tuning success.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
