指令学习范式--模型仅从任务描述中学习执行新任务--在通用模型研究中变得流行。然而,大型Transformer模型作为指令学习者的能力仍然知之甚少。我们使用受控的合成环境来表征这种能力。具体来说,我们使用决定给定字符串是否匹配正则表达式(视为指令)的任务来识别使指令学习具有挑战性的任务、指令和实例的属性。例如,我们发现我们的模型,一个微调的基于T5的text 2 text Transformer,与大型正则语言斗争,这表明不太精确的指令对模型来说是一个挑战。另外,需要跟踪先前步骤的较长上下文的指令执行也更困难。我们使用我们的研究结果系统地构建了一个具有挑战性的指令学习数据集,我们称之为硬RegSet。通过对硬RegSet进行微调,我们的大型Transformer只能正确解释65.6%的测试指令(至少90%的准确率),以及分布外泛化设置中11%-24%的指令。我们提出了硬RegSet作为一个具有挑战性的指令学习任务,和一个可控的环境研究指令学习。
英文摘要:
The instruction learning paradigm -- where a model learns to perform new tasks from task descriptions alone -- has become popular in general-purpose model research. The capabilities of large transformer models as instruction learners, however, remain poorly understood. We use a controlled synthetic environment to characterize such capabilities. Specifically, we use the task of deciding whether a given string matches a regular expression (viewed as an instruction) to identify properties of tasks, instructions, and instances that make instruction learning challenging. For instance, we find that our model, a fine-tuned T5-based text2text transformer, struggles with large regular languages, suggesting that less precise instructions are challenging for models. Additionally, instruction executions that require tracking longer contexts of prior steps are also more difficult. We use our findings to systematically construct a challenging instruction learning dataset, which we call Hard RegSet. Fine-tuning on Hard RegSet, our large transformer learns to correctly interpret only 65.6% of test instructions (with at least 90% accuracy), and 11%-24% of the instructions in out-of-distribution generalization settings. We propose Hard RegSet as a challenging instruction learning task, and a controlled environment for studying instruction learning.
University of Lodz(卢兹大学);NASK National Research Institute(国家研究 institute);Institute of Slavic Studies PAS(波兰科学院斯拉夫研究所);National Information Processing Institute(国家信息处理研究所);Institute of Computer Science PAS(波兰科学院计算机科学研究所);Wroclaw Tech(沃拉日县技术大学)
From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning
Computational Linguistics, Department of Linguistics University of Potsdam(乌特雷赫特大学语言学系计算语言学部);German Research Center for Artificial Intelligence (DFKI), Berlin(德国人工智能研究中心(DFKI)柏林)
ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
Department of Computer Science and Engineering, University of Notre Dame(诺丁汉大学计算机科学与工程系);MIT(麻省理工学院);CalTech(加州理工学院);MBZUAI(穆斯林人工智能研究所);CMU(卡内基梅隆大学);Department of Chemistry & Biochemistry, University of Notre Dame(诺丁汉大学化学与生物化学系)
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
Microsoft Healthcare & Life Sciences(微软医疗与生命科学);Microsoft Research Montréal(微软研究院蒙特利尔);Mila, Université de Montréal(蒙特利尔大学米拉研究所);IKIM, University Hospital Essen(埃森大学医院人工智能医学研究所);Cancer Research Center Cologne Essen(科隆埃森癌症研究中心);German Cancer Consortium(德国癌症联盟);TU Dortmund(多特蒙德工业大学)
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Microsoft(微软);Singapore University of Technology and Design(新加坡科技设计大学);Peking University(北京大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Harbin Institute of Technology(哈尔滨工业大学);School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院)
BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Microsoft(微软);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校);Amazon(亚马逊);Amazon AGI(亚马逊人工智能实验室)