Instruction-tuning plays a vital role in enhancing the task-solving abilities of large language models (LLMs), improving their usability in generating helpful responses on various tasks. However, previous work has demonstrated that they are sensitive to minor variations in instruction phrasing. In this paper, we explore whether introducing perturbations in instruction-tuning data can enhance LLMs' resistance against noisy instructions. We focus on how instruction-tuning with perturbations, such as removing stop words or shuffling words, affects LLMs' performance on the original and perturbed versions of widely-used benchmarks (MMLU, BBH, GSM8K). We further assess learning dynamics and potential shifts in model behavior. Surprisingly, our results suggest that instruction-tuning on perturbed instructions can, in some cases, improve downstream performance. These findings highlight the importance of including perturbed instructions in instruction-tuning, which can make LLMs more resilient to noisy user inputs.
Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Department of Computer Science, School of Computing, Institute of Science Tokyo（计算机科学系、计算学院、科学东京研究所）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）；Research and Development Center for Large Language Models, NII（大型语言模型研究与开发中心、日本科学振兴会）
A Survey on Data Selection for LLM Instruction Tuning
Harbin Institute of Technology（哈尔滨工业大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Harbin Institute of Technology (Weihai)（哈尔滨工业大学（威海））
Instruction tuning is an emergent paradigm in NLP wherein natural language instructions are leveraged with language models to induce zero-shot performance on unseen tasks. Instructions have been shown to enable good performance on unseen tasks and datasets in both large and small language models. Dialogue is an especially interesting area to explore instruction tuning because dialogue systems perform multiple kinds of tasks related to language (e.g., natural language understanding and generation, domain-specific interaction), yet instruction tuning has not been systematically explored for dial…
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
Washington University in St. Louis（华盛顿大学圣路易斯分校）；The University of British Columbia（不列颠哥伦比亚大学）；Google Research（谷歌研究）；Virginia Tech（弗吉尼亚理工大学）；University of California, Davis（加州大学戴维斯分校）；Sony AI（索尼人工智能）；University of California, Berkeley（加州大学伯克利分校）
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Department of Computer Science, School of Computing, Institute of Science Tokyo（东京科学大学计算机科学系）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）；Research and Development Center for Large Language Models, NII（大型语言模型研究与开发中心，日本信息产业技术综合研究所）
CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
CoLing Lab, Department of Philology, Literature and Linguistics, University of Pisa（皮尔切大学哲学、文学与语言学系语言实验室）；Department of Computer Science, University of Pisa（皮尔切大学计算机科学系）
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）；Cohere；Tencent AI Lab（腾讯AI实验室）
PACIT: Unlocking the Power of Examples for Better In-Context Instruction Tuning
Nanjing University（南京大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Southern University of Science and Technology（南方科技大学）；Shanghai University of Finance and Economics（上海财经大学）
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Google（谷歌）；University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Department of Linguistics and Translation, City University of Hong Kong（香港城市大学语言学及翻译学系）
Minimal-Edit Instruction Tuning for Low-Resource Indic GEC
Indian Heritage Language Computing Team Special and Strategic Projects (SSP) Group（印度遗产语言计算团队特殊与战略项目组）；Centre for Development of Advanced Computing (C DAC)（高级计算发展中心）
