发表机构
University of Trieste; McGovern Institute, MIT（的里雅斯特大学; 麻省理工学院麦戈文脑科学硏究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究过参数化模型参数对称性下的 PAC - 贝叶斯分析，通过在商预测器空间构造规范参数化及考虑几何体积，将中性先验转换为反映模型隐式偏差的先验，实验验证其能降低相关指标。
AI 中文摘要
过参数化模型通常具有连续参数对称性，不同参数定义相同预测器。我们表明 PAC - 贝叶斯分析应在商预测器空间上进行，将先验和后验推到商空间可保留经验和总体吉布斯风险，消除仅由相同预测器参数化中两个分布差异引起的非负 KL 贡献。单独求商不能确定使用哪个先验。我们为每个预测器构造一个参数化的规范选择，并考虑其等效参数化的几何体积。这将中性参考先验转换为反映模型隐式偏差的数据独立先验。它近似理想但不可行的后验匹配先验。我们在傅里叶回归和查询 - 键注意力中进行测试，结果表明隐式偏差先验在傅里叶 - 哈达玛实验中降低了平均商空间 KL 的 40.69%和平均 PAC - 贝叶斯证书的 21.40%。
英文摘要
Overparameterized models often have continuous parameter symmetries, so different parameters define the same predictor. We show that PAC--Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs risks while removing the nonnegative KL contribution caused solely by how the two distributions differ among parameterizations of the same predictor. Quotienting alone does not determine which prior to use. We construct a canonical choice of one parameterization for each predictor and account for the geometric volume of its equivalent parameterizations. This transforms a neutral reference prior into a data-independent prior that reflects the model's implicit bias. It approximates the ideal but inadmissible posterior-matched prior, which would minimize the KL term by depending on the training data. The resulting certificate is tighter exactly when this geometry-induced prior has smaller KL divergence from the learned quotient posterior than the neutral prior. We test this prediction in Fourier regression with a Hadamard parameterization and in Query-Key attention, using ordinary SGD without an explicit regularizer. The implicit-bias prior reduces the mean quotient-space KL by \(40.69\%\) and the mean PAC--Bayes certificate by \(21.40\%\) in the Fourier-Hadamard experiment. The smaller, prior-scale-dependent improvement in Query-Key attention confirms the predicted conditional nature of the effect.
