Transformer-based encoder-decoder models produce a fused token-wise representation after every encoder layer. We investigate the effects of allowing the encoder to preserve and explore alternative hypotheses, combined at the end of the encoding process. To that end, we design and examine a $\textit{Multi-stream Transformer}$ architecture and find that splitting the Transformer encoder into multiple encoder streams and allowing the model to merge multiple representational hypotheses improves performance, with further improvement obtained by adding a skip connection between the first and the final encoder layer.
Transformers represent belief state geometry in their residual stream
Simplex（辛普莱克斯）；PIBBSS；Pitzer and Scripps College（皮策与斯克里普斯学院）；University College London（伦敦大学学院）；BITS Principles of Intelligent Behaviour in Biological and Social Systems（BITS生物与社会系统智能行为原理中心）
Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR
Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Decoder-only Streaming Transformer for Simultaneous Translation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）
TransGAT: Transformer-Based Graph Neural Networks for Multi-Dimensional Automated Essay Scoring
Department of Computer Science, King Abdulaziz University（计算机科学系，国王阿卜杜勒阿齐兹大学）；English Language Institute, King Abdulaziz University（英语语言学院，国王阿卜杜勒阿齐兹大学）
