arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.09844cs.LGstat.ML
收藏

通过 Transformers 解决经验贝叶斯问题

Solving Empirical Bayes via Transformers

Department of EECS, MIT(MIT电子工程与计算机科学系)

Anzo Teh, Mark Jabbour, Yury Polyanskiy

AI总结:

本工作提出使用transformers解决经验贝叶斯下的泊松均值估计问题,通过在合成数据上预训练实现上下文学习,理论上证明其遗憾值随维度趋于零,实践中极小模型即超越经典NPMLE算法。

摘要:

本工作将现代AI工具(transformers)应用于解决最古老的统计问题之一:经验贝叶斯(Poisson-EB)设定下的泊松均值。在Poisson-EB中,基于$X=\mathrm{Poisson}(θ)$来估计高维均值向量$θ$(其独立同分布坐标从未知先验$π$中采样)。transformer模型在一组合成生成的$(X,θ)$对上进行预训练,并通过适应未知的$π$来学习进行上下文学习(ICL)。理论上,我们证明了当维度趋于无穷大时,一个足够宽的transformer相对于了解$π$的oracle估计器可以实现趋于零的遗憾值。在实践中,我们发现极小的模型(100k参数)在运行时间和验证损失上均能超越最佳的经典算法(非参数最大似然法,即NPMLE),我们在分布外合成数据及真实世界数据集(NHL hockey、MLB baseball、BookCorpusOpen)上计算了这些指标。最后,通过使用线性探针,我们确认transformer的EB估计器在内部工作机制上似乎不同于NPMLE或Robbins估计器。

英文摘要:

This work applies modern AI tools (transformers) to solving one of the oldest statistical problems: Poisson means under empirical Bayes (Poisson-EB) setting. In Poisson-EB a high-dimensional mean vector $θ$ (with iid coordinates sampled from an unknown prior $π$) is estimated on the basis of $X=\mathrm{Poisson}(θ)$. A transformer model is pre-trained on a set of synthetically generated pairs $(X,θ)$ and learns to do in-context learning (ICL) by adapting to unknown $π$. Theoretically, we show that a sufficiently wide transformer can achieve vanishing regret with respect to an oracle estimator who knows $π$ as dimension grows to infinity. Practically, we discover that already very small models (100k parameters) are able to outperform the best classical algorithm (non-parametric maximum likelihood, or NPMLE) both in runtime and validation loss, which we compute on out-of-distribution synthetic data as well as real-world datasets (NHL hockey, MLB baseball, BookCorpusOpen). Finally, by using linear probes, we confirm that the transformer's EB estimator appears to internally work differently from either NPMLE or Robbins' estimators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑