通过伯努利丢弃理解Transformer编码器-解码器表示
Understanding Transformer Encoder-Decoder Representations through Bernoulli Dropout
AI总结：
本研究通过伯努利丢弃方法探索Transformer编码器-解码器表示，分析稀疏性对模型性能的影响，并在翻译任务中验证了稀疏性阈值对性能的决定性作用。
摘要：
我们通过高维编码器-解码器嵌入中的角度相似性来研究Transformer的过度参数化。我们应用伯努利丢弃在编码器和解码器之间，通过改变保留概率p来确定一个依赖稀疏性的阈值，该阈值之上Top-1预测得以保持。理论上，我们证明，如果有效稀疏嵌入足够大，因此解码器性能在中等坐标丢弃下保持稳定。实验上，我们通过构建一个新增二进制擦除通道（BEC）的新型Transformer模型，并在英语-法语翻译任务上测试其性能。实验结果可视化了验证准确率和BLEU分数的趋势，两者在某些阈值后急剧下降。
英文摘要：
We study Transformer overparameterization through the lens of angular similarity in high-dimensional encoder-decoder embeddings. We apply Bernoulli dropout between the encoder and the decoder, varying the keep probability $p$ to identify a sparsity-dependent threshold above which the Top-1 prediction is preserved. Theoretically, we prove that, if the effective sparsity embeddings is sufficiently large, and thus decoder performance, remain stable under moderate coordinate dropout. Empirically, we implement the Bernoulli dropout by constructing a new Transformer model augmented with Binary Erasure Channel (BEC) and test its performance on an English-French translation task. Experimental results visualize the trends for validation accuracies and BLEU scores, both decline sharply at some threshold.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
