使用特征化低秩混合实现跨语言与跨任务泛化
Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixtures
AI总结：
提出 FLix 参数高效微调方法，通过为语言、任务等数据集特征配置并组合专属低秩更新参数，缓解多语言多任务混合中的负干扰，提升未见数据集上的泛化与零样本性能。
摘要：
将预训练大语言模型（LLM）适配到数十或数百种人类语言中的各类下游任务，在计算上代价高昂。参数高效微调（PEFT）通过仅调整少量参数，显著降低了适配成本。然而，常见的 PEFT 方法 LoRA（Hu et al., 2022）由于激进的参数绑定以及不同数据集之间的负向干扰，在多样化数据集混合上表现欠佳。本文提出 Featurized Low-rank Mixtures（FLix，特征化低秩混合），这是一种为高效多任务多语言适配而设计的新型 PEFT 方法。FLix 将每个独特的数据集特征（例如数据集的语言或任务）与其专属的低秩权重更新参数相关联。通过为每个数据集组合特定于特征的参数，FLix 能够适应多样化的数据集混合，并更好地泛化到未见过的数据集。实验表明，FLix 在监督学习和零样本设置下均能在多种任务上带来显著提升，其中在零样本语义解析中最高获得 14.2 个非精确匹配点的增益。
英文摘要：
Adapting pretrained large language models (LLMs) to various downstream tasks in tens or hundreds of human languages is computationally expensive. Parameter-efficient fine-tuning (PEFT) significantly reduces the adaptation cost, by tuning only a small amount of parameters. However, common PEFT methods LoRA (Hu et al., 2022) suffer from suboptimal performance on diverse dataset mixtures, due to aggressive parameter tying and negative interference among different datasets. In this work, we propose Featurized Low-rank Mixtures (FLix), a novel PEFT method designed for effective multitask multilingual adaptation. FLix associates each unique dataset feature, such as the dataset's language or task, with its own low-rank weight update parameters. By composing feature-specific parameters for each dataset, FLix can accommodate diverse dataset mixtures and generalize better to unseen datasets. Our experiments show that FLix leads to significant improvements over a variety of tasks for both supervised learning and zero-shot settings with gains of up to $14.2$ inexact match points in zero-shot semantic parsing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
