噪声会误导针对稀疏目标的旋转不变算法
Noise misleads rotation invariant algorithms on sparse targets
AI总结：
该研究指出噪声环境下旋转不变算法在稀疏线性问题中即使样本量达维度仍非最优，通过贝叶斯最优下界、非旋转不变算法上界及梯度流轨迹分析验证结论，为算法设计与下界分析提供新方法。
摘要：
众所周知，当样本数量低于问题的“维度”时，即便用于学习稀疏线性问题，旋转不变算法族的表现也未达最优。该算法族包含任何采用全连接输入层（以旋转对称分布初始化）、通过梯度下降训练的神经网络。最简单的稀疏问题是从$d$个特征中学习单个特征。在该场景下，分类误差或回归损失随$1-k/n$增长，其中$k$为已观测的样本数量。当样本数量$k$达到维度$d$时，这些下界将失去意义。\n 我们证明，当该稀疏线性问题加入噪声后，旋转不变算法在观测到$d$个或更多样本后仍未达最优。我们通过对旋转对称化问题的贝叶斯最优算法推导下界，证明了这一结论。随后，我们针对同一问题，证明了简单非旋转不变算法的上界要低得多。最后，我们在若干简单场景下分析了多种标准优化算法的梯度流轨迹，展示了它们朝向或偏离稀疏目标的变化方式。\n 我们认为，我们的轨迹分类方法将有助于设计能够利用稀疏目标的算法，而我们证明下界的方法对于分析具备其他不变性类别 的算法族至关重要。
英文摘要：
It is well known that the class of rotation invariant algorithms are suboptimal even for learning sparse linear problems when the number of examples is below the "dimension" of the problem. This class includes any gradient descent trained neural net with a fully-connected input layer (initialized with a rotationally symmetric distribution). The simplest sparse problem is learning a single feature out of $d$ features. In that case the classification error or regression loss grows with $1-k/n$ where $k$ is the number of examples seen. These lower bounds become vacuous when the number of examples $k$ reaches the dimension $d$. We show that when noise is added to this sparse linear problem, rotation invariant algorithms are still suboptimal after seeing $d$ or more examples. We prove this via a lower bound for the Bayes optimal algorithm on a rotationally symmetrized problem. We then prove much lower upper bounds on the same problem for simple non-rotation invariant algorithms. Finally we analyze the gradient flow trajectories of many standard optimization algorithms in some simple cases and show how they veer toward or away from the sparse targets. We believe that our trajectory categorization will be useful in designing algorithms that can exploit sparse targets and our method for proving lower bounds will be crucial for analyzing other families of algorithms that admit different classes of invariances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
