StructLM：面向构建结构化知识接地通用模型的研究
StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
AI总结：
针对大语言模型结构化知识接地能力不足的问题，研究构建百万级指令微调数据集训练StructLM系列模型，在多数任务上超越专用模型并实现强泛化，同时发现规模扩大收益有限。
摘要：
结构化数据源（如表格、图和数据库）是无处不在的知识来源。尽管大语言模型（LLM）在纯文本任务上已展现出强大能力，但它们在解读和利用结构化数据方面的熟练度仍然有限。我们的研究发现，LLM在处理结构化数据的能力上存在显著缺陷，例如ChatGPT的表现平均落后于当前最优（SoTA）模型35%。为增强LLM的结构化知识接地（Structured Knowledge Grounding，SKG）能力，我们构建了一个包含110万个样本的综合指令微调数据集。利用该数据集，我们基于Mistral和CodeLlama模型家族训练了一系列参数规模从7B到34B不等的模型，命名为StructLM。我们的StructLM系列在18个评估数据集中的16个上超越了任务特定模型，并在8项SKG任务上创下新的SoTA性能。此外，StructLM在6项未见过的新型SKG任务上展现出强泛化能力，平均优于TableLlama 35%，平均优于Flan-UL2 20B 10%。与预期相反，我们观察到模型规模扩大带来的收益有限，StructLM-34B仅比StructLM-7B有小幅提升。这表明结构化知识接地仍然是一项具有挑战性的任务，需要更具创新性的设计才能推动其达到新水平。
英文摘要：
Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
