Transformers learn factored representations
Transformer 学习因式分解表示
机构 * Simplex, Astera Institute ; Sussex Centre for Consciousness Science, University of Sussex ; Centre for Complexity Science ; Center for Psychedelic Research, Department of Brain Sciences, Imperial College London ; Center for Eudaimonia ; Human Flourishing, University of Oxford ; Beyond Institute for Theoretical Science (BITS)
AI总结 Transformer通过预训练学习将世界分解为部分,并在正交子空间中表示因素,即使在条件独立性被破坏时仍倾向于保持这种因式分解表示。