arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.06045cs.CLcs.LG
收藏

一个通用问答模型的结构分析

Structural analysis of an all-purpose question answering model

University of Geneva(日内瓦大学);Illuin Technology(Illuin科技公司)

Vincent Micheli, Quentin Heinrich, François Fleuret, Wacim Belblidia

摘要:

注意力是现在普遍存在的预训练语言模型的关键组成部分。通过学习专注于相关的信息,这些基于transformer的架构已经证明能够同时处理多个任务,有时甚至超过单任务。为了更好地理解这种现象,我们对我们引入的新的通用问答模型进行了结构分析。令人惊讶的是,即使在任务之间没有强烈的转移效应的情况下,该模型也保留了单任务性能。通过注意头的重要性评分,我们观察到,注意头专门在一个特定的任务,一些头比其他人更有利于学习在多任务和单任务设置。

英文摘要:

Attention is a key component of the now ubiquitous pre-trained language models. By learning to focus on relevant pieces of information, these Transformer-based architectures have proven capable of tackling several tasks at once and sometimes even surpass their single-task counterparts. To better understand this phenomenon, we conduct a structural analysis of a new all-purpose question answering model that we introduce. Surprisingly, this model retains single-task performance even in the absence of a strong transfer effect between tasks. Through attention head importance scoring, we observe that attention heads specialize in a particular task and that some heads are more conducive to learning than others in both the multi-task and single-task settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑