Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Microsoft（微软）；Singapore University of Technology and Design（新加坡科技设计大学）；Peking University（北京大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China（中国科学院软件研究所中文信息处理实验室）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）
TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc（百度公司）
From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning
Computational Linguistics, Department of Linguistics University of Potsdam（乌特雷赫特大学语言学系计算语言学部）；German Research Center for Artificial Intelligence (DFKI), Berlin（德国人工智能研究中心（DFKI）柏林）
ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
Department of Computer Science and Engineering, University of Notre Dame（诺丁汉大学计算机科学与工程系）；MIT（麻省理工学院）；CalTech（加州理工学院）；MBZUAI（穆斯林人工智能研究所）；CMU（卡内基梅隆大学）；Department of Chemistry & Biochemistry, University of Notre Dame（诺丁汉大学化学与生物化学系）
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
Microsoft Healthcare & Life Sciences（微软医疗与生命科学）；Microsoft Research Montréal（微软研究院蒙特利尔）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；IKIM, University Hospital Essen（埃森大学医院人工智能医学研究所）；Cancer Research Center Cologne Essen（科隆埃森癌症研究中心）；German Cancer Consortium（德国癌症联盟）；TU Dortmund（多特蒙德工业大学）
IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Nilekani Centre at AI4Bharat（AI4Bharat的Nilekani中心）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；IT University of Copenhagen（丹麦技术大学）；Microsoft（微软）
Evaluating and Aligning CodeLLMs on Human Preference
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Research Center for Social Computing and Interactive Robotics（社会计算与交互机器人研究中心）；Harbin Institute of Technology（哈尔滨工业大学）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
