arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.01749cs.CL
收藏

用于分层自适应的混合效果Transformers

Mixed-effects transformers for hierarchical adaptation

Stanford University(斯坦福大学);Princeton University(普林斯顿大学)

Julia White, Noah Goodman, Robert Hawkins

摘要:

语言的使用在不同的语境中有很大的不同。在某种程度上,像GPT-3这样的现代语言模型能够通过对先前输入文本或提示符的字符串进行调节来解释这种变化。然而,提示是无效的,当上下文是稀疏的,样本外,或extra-textual的,例如,占何时何地产生的文本或谁生产it. In本文中,我们介绍了混合效果的Transformer(MET),一种新的方法学习层次结构的前缀-轻量级模块前置到输入-占结构的变化。具体来说,我们展示了如何流行的混合效应模型类可以扩展到基于变换器的架构,使用正则化的前缀调整过程与辍学。我们在几个领域适应基准上评估了这种方法,发现它可以有效地适应新的环境,同时仍然有效地推广到看不见的环境。

英文摘要:

Language use differs dramatically from context to context. To some degree, modern language models like GPT-3 are able to account for such variance by conditioning on a string of previous input text, or prompt. Yet prompting is ineffective when contexts are sparse, out-of-sample, or extra-textual; for instance, accounting for when and where the text was produced or who produced it. In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes -- lightweight modules prepended to the input -- to account for structured variation. Specifically, we show how the popular class of mixed-effects models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout. We evaluate this approach on several domain-adaptation benchmarks, finding that it efficiently adapts to novel contexts with minimal data while still effectively generalizing to unseen contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑