Transformer中的方向优化不对称性：一种合成压力测试
Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test
AI总结：
本研究通过合成压力测试揭示Transformer在方向优化上的不对称性，发现即使从头训练的GPT-2模型也表现出显著的方向优化差距。
摘要：
Transformer在理论上是反向不变的：其函数类不偏好从左到右的映射。然而，对自然语言的实证研究却反复报告一种“反向诅咒”，而最近关于大语言模型时间不对称性的研究表明，现实世界语料库本身具有自己的时间箭头。这留下了一个未解决的问题：方向性失败是源于语言统计特性，还是源于架构本身？我们通过一个完全合成的、熵受控的基准测试来消除这种歧义，该测试设计为一个干净房间的压力测试，用于方向学习。通过随机字符串映射，可调节分支因子K，我们构建了零条件熵的正向任务和具有分析确定熵地板的反向任务。超过这些地板的损失揭示，即使经过从头训练的GPT-2模型也表现出强烈的、可重复的方向优化差距（例如，在K=5时为1.16纳特），远大于在相同数据上训练的MLP。预训练的初始设置会改变优化行为，但不会消除这一差距，而LoRA在高熵反向映射上遇到一个尖锐的容量墙。这些结果隔离了因果Transformer训练中固有的、与语义无关的方向摩擦的最小特征——即使在去除语言先验、令牌频率和语料库层面时间不对称性时，这一特征仍然存在。我们的基准提供了一种受控的工具，用于解剖现代序列模型中的方向偏差，并激发对为什么反转仍对Transformer根本更难的更深入机理研究。
英文摘要：
Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings. Yet empirical studies on natural language repeatedly report a "reversal curse," and recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time. This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself? We cut through this ambiguity with a fully synthetic, entropy-controlled benchmark designed as a clean-room stress test for directional learning. Using random string mappings with tunable branching factor K, we construct forward tasks with zero conditional entropy and inverse tasks with analytically determined entropy floors. Excess loss above these floors reveals that even scratch-trained GPT-2 models exhibit a strong, reproducible directional optimization gap (e.g., 1.16 nats at K=5), far larger than that of an MLP trained on the same data. Pre-trained initializations shift optimization behavior but do not eliminate this gap, while LoRA encounters a sharp capacity wall on high-entropy inverse mappings. Together, these results isolate a minimal, semantics-free signature of directional friction intrinsic to causal Transformer training-one that persists even when linguistic priors, token frequencies, and corpus-level temporal asymmetries are removed. Our benchmark provides a controlled instrument for dissecting directional biases in modern sequence models and motivates deeper mechanistic study of why inversion remains fundamentally harder for Transformers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
