AI 中文总结
该研究发现，深度强化学习中冻结的随机CNN特征提取器训练的智能体会自发形成稀疏全连接表示，稀疏度随任务复杂度变化，活跃神经元对性能至关重要，还揭示了无显式稀疏机制下的问题有效秩。
AI 中文摘要
我们报告了一个引人注目的现象：使用冻结的、随机初始化的CNN特征提取器训练的深度强化学习智能体，在没有任何稀疏性诱导目标的情况下，自发形成了极其稀疏的全连接表示。在第一个全连接层（FC1，3136→64）中，对于确定性Pong游戏，智能体通过64个神经元中仅1-3个来压缩任务相关信息（随机Pong游戏为5-11个），而在匹配条件下，可训练CNN会激活55-64个神经元。我们确立了四个主要发现：第一，FC1稀疏性随任务复杂度变化：Pong为1-11，Breakout为19-26，Space Invaders约为42；宽度缩放证实这反映了任务结构，而非固定容量比例。第二，游戏内缩放现象出现：三个相同的Pong随机种子分别产生5、7、11个活跃神经元；5个神经元的种子在奖励+14时达到平台，其余则达到专家性能（+18.4、+18.7），表明随机投影的可用维度限制了可实现的性能。第三，消融实验证实其必要性：移除这些活跃神经元会导致两种PPO实现及四款游戏的性能崩溃。第四，信息瓶颈早早就确定了：一次搜索显示活跃集在15-30M步时锁定，而奖励在35-105M步后转为正值。在Breakout中的补充发现表明，冻结和可训练CNN通过结构不同的瓶颈达到了可比奖励：冻结智能体使用17-25个活跃神经元（参与率约10-14），而可训练智能体使用51个（参与率约3.6）。最后，当输入维度远大于内在任务维度时，对冻结随机投影的梯度下降可能会揭示潜在问题的有效秩，而无需明确的稀疏性机制。
英文摘要
We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective. In the first fully-connected layer (FC1, $3{,}136 \to 64$), agents compress task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong (5-11 for stochastic Pong), while trainable CNNs activate 55-64 neurons under matched conditions. We establish four principal findings. First, FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and $\sim$42 for Space Invaders. Width-scaling confirms this reflects task structure rather than a fixed capacity fraction. Second, within-game scaling emerges: three identical Pong seeds produce 5, 7, and 11 active neurons. The 5-neuron seed plateaus at $+14$ reward, while the others reach expert performance ($+18.4$, $+18.7$), suggesting the random projection's usable dimensionality bounds achievable performance. Third, ablation confirms necessity: removing these active neurons crashes performance across two PPO implementations and four games. Fourth, the information bottleneck commits early: a sweep shows the active set locks by 15-30M steps, while reward turns positive 35-105M steps later. A complementary finding in Breakout shows frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio $\sim$10-14), while trainable agents use 51 (participation ratio $\sim$3.6). Finally, wherever input dimensionality dwarfs intrinsic task dimensionality, gradient descent on a frozen random projection may reveal the effective rank of the underlying problem without explicit sparsity machinery.
Comments24 pages, 4 figures
