SCITUNE：对齐大型语言模型与人类整理的科学多模态指令
SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
AI总结：
SCITUNE通过多模态指令提升大语言模型在科学任务中的表现，优于现有模型并在ScienceQA基准上超越人类性能。
摘要：
指令微调是一种流行的对齐大型语言模型（LLM）与人类意图的范式。尽管其受欢迎，这一理念在改进LLM以对齐现有基础模型与科学学科、概念和目标方面较少被探索。本文提出SCITUNE作为微调框架，以提升LLM遵循科学出版物生成的多模态指令的能力。为测试我们的方法，我们训练了一个大型多模态模型LLaMA-SciTune，该模型结合了视觉编码器和LLM用于科学聚焦的视觉和语言理解。LLaMA-SciTune在生成的图表类型和描述上显著优于现有最先进模型，在SciCap和VisText基准中表现突出。与仅使用合成数据微调的模型相比，LLaMA-SciTune在ScienceQA基准上平均和许多子类别中超越了人类表现。我们的结果表明，尽管其体积和相对稀缺性低于合成数据，人类生成的科学多模态指令在微调LLM以在科学任务中表现良好方面仍然具有高度价值。我们公开发布了SciTune代码库https://github.com/pnnl/scitune。
英文摘要：
Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present \textit{SciTune} as a tuning framework to improve the ability of LLMs to follow multimodal instructions generated from scientific publications. To test our methodology, we train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in SciCap and VisText benchmarks. In comparison to the models that are finetuned with synthetic data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark. Our results demonstrate that human-generated scientific multimodal instructions remain highly valuable in tuning LLMs to perform well on science tasks, despite their lower volume and relative scarcity compared to synthetic data. We publicly release the SciTune codebase https://github.com/pnnl/scitune.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
