arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.21731cs.LG
收藏

先验数据拟合网络中光谱结构的机理证据

Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks

Indian Institute of Technology Roorkee (IIT Roorkee)(印度理工学院罗尔基分校)

Kaustubh Sharma, Srijan Tiwari, Ojasva Nema, Parikshit Pareek

AI总结:

本文通过实验和解码器验证了先验数据拟合网络学习了结构化的光谱表示,这些表示可作为显式核提取,展示了其在连续回归任务中的涌现特性。

摘要:

先验数据拟合网络(PFNs)能够在单次前向传递中实现近似贝叶斯推断,但其内部表示仍不明晰。本文提供了机理证据表明PFNs学习了结构化的光谱表示,这些表示可被提取为显式核。首先,通过对三种架构的探测实验显示,潜变量注意力分数中的光谱信息可线性解码并沿主导主轴组织。激活块状和目标子空间干预实验表明,这种信息用于预测并集中在低维子空间中,光谱方向比随机方向更有效。关键的是,这些性质在TabPFN上对合成异常输入和现实时间序列(如航空乘客和牛奶生产)均成立,表明这些特性是PFN风格连续回归任务近似中的涌现特征,而非训练先验的产物。其次,本文引入了滤波器银行解码器,将冻结的PFN潜变量映射到显式光谱密度,通过Bochner定理重建平稳核。所得核支持与迭代基线相媲美且仅需单次前向传递的GP回归,证明PFN先验不仅是隐含的,还可作为可移植的贝叶斯对象显式恢复。

英文摘要:

Prior-Data Fitted Networks (PFNs) perform approximate Bayesian inference in a single forward pass, and tabular foundation models (TFMs) built on them are now widely used. To understand what networks infer internally, recent mechanistic studies of TFMs locate where predictions form, but treat these models as tabular predictors rather than as PFNs. It therefore remains unknown whether PFNs represent the spectral content of their context, the quantity that specifies a stationary kernel, and whether this content can be read out as an explicit kernel. We answer both questions. First, across seven PFNs, including four pretrained TFMs and a model trained only on a decision-tree prior, a linear probe on the residual stream recovers the frequency of the context with $R^2 \geq 0.95$. This structure is led by a single principal direction. Second, activation and subspace patching show that the network uses the structure through a low-dimensional subspace, where a few spectral directions move predictions far more than random ones. This holds even for the decision-tree model, so a spectral training prior is not required. On real datasets with up to 499 features, 64 of the 192 directions of TabPFN, chosen without labels, carry 85 to 95\% of the causal effect of the context in all but one pair. Third, we introduce a Filter Bank Decoder that turns frozen PFN representations into an explicit stationary kernel through Bochner's theorem. Without any test-time optimization, the decoded kernel supports Gaussian process regression competitive with deep kernel learning and random Fourier features at about $200\times$ lower cost. PFN latents therefore hold spectral structure that is causally used and recoverable as a portable kernel.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑