无传递的力：一种由深度引起的秩坍缩，任何表示上的损失都无法重新打开
Force without transmission: a depth-induced rank collapse that no loss on the representation reopens
浏览论文内容
中文总结 AI 辅助
本研究探究Transformer秩坍缩的修复机制，发现修复取决于梯度是否到达需改变的权重（如查询键权重），而非损失项强度；恢复跳跃连接可重新打开路径，但损失仍高于健康网络。
中文摘要 AI 辅助
训练可以将Transformer推向秩坍缩：所有token表示指向一个方向，学习停止。在一种相关的注意力坍缩中，具有有界修正力的损失项在运行期间修复了网络。我们询问这样的项是否能修复秩坍缩。我们通过削弱跳跃连接来坍缩小型Transformer，并处理坍缩网络的副本。没有添加的损失项修复了坍缩，尽管更强的类型以任务梯度的约十分之一推动。原因是路径，而非强度。任务梯度不再到达决定注意力位置的查询和键权重，而添加项的梯度在坍缩形成的块之前就消失了。恢复跳跃连接（不改变任何权重）立即重新打开了这条路径。秩随后恢复，但仅在坍缩尺度之上。在高学习率的爆发之后，路径保持打开，秩未经处理就恢复了。来自路径的注册预测对恢复时间进行了排序，但并未转移到这一原因。在每种情况下，秩恢复后损失仍高于健康网络的损失。坍缩网络能否被修复取决于梯度是否仍然到达必须改变的权重，而不是损失项推动的强度。
英文摘要
Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops. In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run. We ask whether such a term repairs rank collapse. We collapse small transformers by weakening their skip connection and treat copies of the collapsed network. No added loss term repaired the collapse, although the stronger kind pushed with about a tenth of the task gradient. The reason was the path, not the strength. The task gradient no longer reached the query and key weights, which decide where attention looks, and the added term's gradient faded before the blocks where the collapse forms. Restoring the skip connection, which changes no weight, reopened this path at once. The rank then recovered, but only far above the scale of collapse. After a burst of high learning rate the path stayed open and the rank recovered untreated. Registered predictions from the path ranked recovery times but did not transfer to this cause. In every case the loss stayed above that of a healthy network after the rank recovered. Whether a collapsed network can be repaired depends on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes.
发表机构
- Technische Universität Ilmenau（伊尔默瑙工业大学）
- German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig（德国综合生物多样性研究中心（iDiv）哈勒-耶拿-莱比锡）
- Friedrich Schiller University（弗里德里希·席勒大学）
机构由 AI 辅助整理，请以论文原文为准。
