An Empirical Study of Instruction-tuning Large Language Models in Chinese
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；APUS AiLMe Lab（APUS AiLMe实验室）
The success of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open-source community's interest in instruction-tuning, which is deemed to accelerate ChatGPT's replication process. However, research on instruction-tuning LLMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that provides valuable findings for effectively customizing LLMs that can better respond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most important elements for instruction-tuning. Besides, we also conduct experiment to study the impact of other factors, e.g., chain-of-thought data and human-value alignment. We hope that this empirical study can make a modest contribution to the open Chinese version of ChatGPT. This paper will release a powerful Chinese LLMs that is comparable to ChatGLM. The code and data are available at https://github.com/PhoebusSi/Alpaca-CoT.
Chinese Open Instruction Generalist: A Preliminary Release
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The University of Sheffield（谢菲尔德大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；Dartmouth University（达特茅斯学院）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）；Carnegie Mellon University（卡内基梅隆大学）
COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；M-A-P；Institute of Automation, CAS（中国科学院自动化研究所）；Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
HKUST（香港科技大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Ohio State University（俄亥俄州立大学）；AIWaves Inc.（AIWaves公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；Peking University（北京大学）；Beijing Foreign Studies University（北京外国语大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）
CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare
School of Software Engineering（软件工程学院）；University of Science and Technology of China（中国科学技术大学）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；Xiangshui County Party School（响水县党校）；School of Biomedical Engineering（生物医学工程学院）；UNSW（新南威尔士大学）
Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
Pretrained Language Models (PLMs) have achieved tremendous success in natural language understanding tasks. While different learning schemes -- fine-tuning, zero-shot, and few-shot learning -- have been widely explored and compared for languages such as English, there is comparatively little work in Chinese to fairly and comprehensively evaluate and compare these methods and thus hinders cumulative progress. In this paper, we introduce the Chinese Few-shot Learning Evaluation Benchmark (FewCLUE), the first comprehensive few-shot evaluation benchmark in Chinese. It includes nine tasks, ranging …
Demystifying Instruction Mixing for Fine-tuning Large Language Models
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；LibrAI；Monash University（莫纳什大学）；University of British Columbia（不列颠哥伦比亚大学）；The University of Melbourne（墨尔本大学）
No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks
Yunnan University（云南大学）；The University of Manchester（曼彻斯特大学）；Wuhan University（武汉大学）；University of Florida（佛罗里达大学）；The Chinese University of Hong Kong（香港中文大学）；The Fin AI
ZhiFangDanTai: Fine-tuning Graph-based Retrieval-Augmented Generation Model for Traditional Chinese Medicine Formula
School of Management, Capital Normal University（管理学院，首都师范大学）；School of Electrical Engineering and Computer Science, the University of Queensland（电气工程与计算机科学学院，昆士兰大学）
Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuning
College of Computer Science and Technology, Jilin University, China（吉林大学计算机科学与技术学院）；Key Laboratory of Ancient Chinese Script, Culture Relics and Artificial Intelligence, Jilin University, China（吉林大学古文字、文物与人工智能重点实验室）；School of Artificial Intelligence, Jilin University, China（吉林大学人工智能学院）；School of Archaeology, Jilin University, China（吉林大学考古学院）
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory（协调科学实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Mathematics（数学系）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Department of Linguistics and Translation, City University of Hong Kong（香港城市大学语言学及翻译学系）
Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning
School of Cyber Science and Engineering Nanjing University of Science and Technology China（南京理工大学信息科学与工程学院）；Meituan China（美团）；Zhipu.AI China（智谱AI）；Institute of Information Engineering Chinese Academy of Sciences China（中国科学院信息工程研究所）
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
DataArc Tech Ltd.（DataArc科技有限公司）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；IDEA Research, International Digital Economy Academy（IDEA研究所，国际数字经济学院）；Hithink RoyalFlush Information Network Co., Ltd（Hithink RoyalFlush信息网络有限公司）
