This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance. Motivated by structured human education, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method was extensively evaluated across model architectures and scales on LongBench and MMedBench datasets, demonstrating superior performance against other knowledge injection methods. We also explored our method's scalability across different training corpus sizes, laying the foundation to enhance domain-specific LLMs with better data utilization.
Plug-and-Play Knowledge Injection for Pre-trained Language Models
Renmin University of China(中国人民大学);Tsinghua University(清华大学);International Innovation Center of Tsinghua University(清华大学国际创新中心);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tencent Inc(腾讯公司);Quan Cheng Laboratory(泉城实验室)
Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
Faculdade de Engenharia Elétrica e de Computação (FEEC), University of Campinas (Unicamp)(电子工程与计算学院(FEEC),坎皮纳斯大学);Instituto de Computação (IC), University of Campinas (Unicamp)(计算研究所(IC),坎皮纳斯大学)
SLiNT: Structure-aware Language Model with Injection and Contrastive Training for Knowledge Graph Completion
University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);National Astronomical Observatories, Chinese Academy of Sciences(中国科学院国家天文台)
Enhancing Chest X-ray Classification through Knowledge Injection in Cross-Modality Learning
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Westlake University Research Center for Industries of the Future(西湖大学未来产业研究中心);The Second Affiliated Hospital Zhejiang University School of Medicine (SAHZU)(浙江大学医学院附属第二医院);Google Developer Relations Team(谷歌开发者关系团队)
RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学可信自主系统研究院和计算机科学与工程系);School of Computer Science, University of Nottingham Ningbo China(宁波大学计算机学院)
Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);AI Research(360人工智能研究院)
Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection
School of Cyber Science Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Wuhan AI Research Foundation Model Research Center(武汉人工智能研究基金会大模型研究中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室)
SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection
img/wse-cube-no-shadow.png Web \& Software Engineering (WSE) Research Group, Leipzig University of Applied Sciences (HTWK Leipzig), Leipzig, Germany CBZ München GmbH, Heilbronn, Germany
SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models
School of Information Engineering, Yangzhou University(扬州大学信息工程学院);Jiangsu Province Engineering Research Center of Knowledge Management and Intelligent Service(江苏省知识管理与智能服务工程研究中心);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系)
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Singapore Management University (SMU)(新加坡管理大学);Nanyang Technological University (NTU)(南洋理工大学);National University of Singapore (NUS)(新加坡国立大学)
When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能全国重点实验室(北京通用人工智能研究院));Shandong University(山东大学);Nanjing University(南京大学);Wuhan University(武汉大学);The University of Sydney(悉尼大学);Northeast Forestry University(东北林业大学);Beijing Institute of Technology(北京理工大学)
KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
University of Science;State Key Laboratory of General Artificial Intelligence, BIGAI;Xiamen University;Northeast Forestry University;Beijing Institute of Technology;Ludwig Maximilian University of Munich;The University of Sydney;C-FAIR\&school of software, Shandong University;State Key Lab. for Novel Software Technology, Nanjing University, P.R. China