arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.00185cs.CL
收藏

Zemi:从多任务中学习零样本半参数化语言模型

Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks

UIUC(伊利诺伊大学厄巴纳-香槟分校);Tencent AI Lab(腾讯人工智能实验室)

Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, Heng Ji

AI总结:

本文提出零样本半参数化语言模型Zemi,通过大规模无标注语料库检索增强多任务提示训练,并引入augmentation fusion模块融合噪声检索结果,在模型规模缩小3.9倍的情况下,于七项任务上超越T0-3B达16%。

摘要:

尽管大型语言模型已展现出令人印象深刻的零样本能力,但巨大的模型规模通常会带来高昂的成本。近期,通过外部检索器增强较小语言模型的半参数化语言模型,展现出了有前景的语言建模能力。然而,此类半参数化语言模型在下游任务的零样本泛化上,能否表现得与其全参数化对应模型一样出色,仍不明确。在本研究中,我们引入了Zemi,一个零样本半参数化语言模型。据我们所知,这是首个能在广泛的留出未见任务上展现出强大零样本性能的半参数化语言模型。我们采用一种新颖的半参数化多任务提示训练范式来训练Zemi,与T0提出的参数化多任务训练相比,该范式显示出显著提升。具体而言,我们通过从大规模任务无关无标注语料库中进行检索,来增强多任务训练和零样本评估。为了整合多个可能带有噪声的检索增强,我们进一步提出了一种新颖的augmentation fusion模块,利用perceiver resampler和门控交叉注意力。值得注意的是,我们提出的Zemi_LARGE在所有七个评估任务上超越了T0-3B 16%,同时模型规模小了3.9倍。

英文摘要:

Although large language models have achieved impressive zero-shot ability, the huge model size generally incurs high cost. Recently, semi-parametric language models, which augment a smaller language model with an external retriever, have demonstrated promising language modeling capabilities. However, it remains unclear whether such semi-parametric language models can perform competitively well as their fully-parametric counterparts on zero-shot generalization to downstream tasks. In this work, we introduce $\text{Zemi}$, a zero-shot semi-parametric language model. To our best knowledge, this is the first semi-parametric language model that can demonstrate strong zero-shot performance on a wide range of held-out unseen tasks. We train $\text{Zemi}$ with a novel semi-parametric multitask prompted training paradigm, which shows significant improvement compared with the parametric multitask training as proposed by T0. Specifically, we augment the multitask training and zero-shot evaluation with retrieval from a large-scale task-agnostic unlabeled corpus. In order to incorporate multiple potentially noisy retrieved augmentations, we further propose a novel $\text{augmentation fusion}$ module leveraging perceiver resampler and gated cross-attention. Notably, our proposed $\text{Zemi}_\text{LARGE}$ outperforms T0-3B by 16% on all seven evaluation tasks while being 3.9x smaller in model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑