The high cost of full-parameter fine-tuning (FFT) of Large Language Models (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods. However, it remains unclear which methods provide the best cost-performance trade-off at different model scales. We introduce Astraios, a suite of 28 instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up to 16 billion parameters. Through investigations across 5 tasks and 8 different datasets encompassing both code comprehension and code generation tasks, we find that FFT generally leads to the best downstream performance across all scales, and PEFT methods differ significantly in their efficacy based on the model scale. LoRA usually offers the most favorable trade-off between cost and performance. Further investigation into the effects of these methods on both model robustness and code security reveals that larger models tend to demonstrate reduced robustness and less security. At last, we explore the relationships among updated parameters, cross-entropy loss, and task performance. We find that the tuning effectiveness observed in small models generalizes well to larger models, and the validation loss in instruction tuning can be a reliable indicator of overall downstream performance.
Demystifying Instruction Mixing for Fine-tuning Large Language Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);LibrAI;Monash University(莫纳什大学);University of British Columbia(不列颠哥伦比亚大学);The University of Melbourne(墨尔本大学)
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory(协调科学实验室);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室)
An Empirical Study of Instruction-tuning Large Language Models in Chinese
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);APUS AiLMe Lab(APUS AiLMe实验室)
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)
On the Effect of Instruction Tuning Loss on Generalization
Dept. of Electrical Engineering(电气工程系);Indian Institute of Technology Delhi(印度理工学院德里);Media and Data Science Research(媒体与数据科学研究);Adobe Inc., India(Adobe公司,印度)
Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning
School of Cyber Science and Engineering Nanjing University of Science and Technology China(南京理工大学信息科学与工程学院);Meituan China(美团);Zhipu.AI China(智谱AI);Institute of Information Engineering Chinese Academy of Sciences China(中国科学院信息工程研究所)
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
DataArc Tech Ltd.(DataArc科技有限公司);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));IDEA Research, International Digital Economy Academy(IDEA研究所,国际数字经济学院);Hithink RoyalFlush Information Network Co., Ltd(Hithink RoyalFlush信息网络有限公司)
University of Southern California(南加州大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);The Pennsylvania State University(宾夕法尼亚州立大学);University of California, Davis(加利福尼亚大学戴维斯分校)
COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);M-A-P;Institute of Automation, CAS(中国科学院自动化研究所);Peking University(北京大学);University of Science and Technology of China(中国科学技术大学);HKUST(香港科技大学);University of Manchester(曼彻斯特大学);University of Waterloo(滑铁卢大学);Vector Institute(向量研究所)