指令微调的统一因果视角
A Unified Causal View of Instruction Tuning
AI总结：
本文提出元结构因果模型（meta-SCM）与结构指令微调（SIT）方法，以学习任务所需的因果因子进行预测，从而提升指令微调在未见任务上的零样本能力。
摘要：
对混合任务的指令微调已经提升了自然语言处理（NLP）中的零样本能力。然而，现有方法通常学习的是指令格式化样本与目标标签之间表现出相关性的特征，而非因果关系。这种相关性在统计学中被称为“虚假相关”，在新任务中可能发生剧烈变化，使得从学习到的特征产生的影响具有误导性。为此，我们开发了一个元结构因果模型（meta-SCM），将不同的NLP任务整合在单一的数据因果结构下。具体来说，meta-SCM引入了多个表示源上下文属性的潜在因子，其中只有部分因子因果性地影响特定任务的目标标签。关键思想是学习任务所需的因果因子，并且仅使用这些因子来对给定任务进行预测。理论上，我们证明了因果因子可以在不与其他信息混合的情况下被识别。在可识别性的指导下，我们提出了一种结构指令微调（SIT）方法，以学习能够模拟每个任务因果因子的任务所需因果表示。我们方法的实用性通过在多种未见过的数据集和任务上零样本能力的提升得到了验证。
英文摘要：
Instruction tuning on a mixture of tasks has improved zero-shot capabilities in natural language processing (NLP). Nevertheless, existing methods often learn features that exhibit correlations between instruction-formatted samples and target labels, rather than causal relationships. Termed as ``spurious correlation'' in statistics, such a correlation may change drastically in a new task, making the effect from the learned features to be misleading. To this end, we develop a meta Structural Causal Model (meta-SCM) to integrate different NLP tasks under a single causal structure of the data. Specifically, the meta-SCM introduces multiple latent factors that represent properties of source context, only some of which causally influence the target labels for a specific task. The key idea is to learn task-required causal factors and only use those to make predictions for a given task. Theoretically, we prove the causal factor can be identified without mixing information from others. Guided by the identifiability, we propose a Structural Instruction Tuning (SIT) method to learn the task-required causal representations that can mimic the causal factors for each task. The utility of our approach is verified by improvements of zero-shot ability on a range of unseen datasets and tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
