Talking-Heads注意
Talking-Heads Attention
Google（谷歌）
摘要：
我们引入了“说话头注意力”--一种多头注意力的变体，它包括在softmax操作之前和之后的注意头维度上的线性投影。虽然只插入了少量的额外参数和适量的额外计算，但说话头注意力在掩蔽语言建模任务中导致了更好的困惑，以及在将学习转移到语言理解和问答任务时更好的质量。
英文摘要：
We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
