arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.10573cs.LGcs.AI
收藏

Transformer中的隐式统计推断:在上下文中近似似然比检验

Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context

Faris Chaudhry, Siddhant Gadkari

AI总结:

Transformer通过构建任务自适应的统计估计器实现上下文学习,而非简单相似性匹配。

摘要:

在上下文学习(ICL)中,Transformer能够适应新任务而无需权重更新,但其底层算法仍缺乏深入理解。我们从统计决策理论的角度出发,研究简单的二元假设检验,其中最优策略由似然比检验决定。值得注意的是,这种设置为机制可解释性提供了数学严谨的环境,其中目标算法的真实值已知。通过训练需要不同几何结构的任务(线性位移均值与非线性方差估计),我们证明模型在上下文中近似从上下文中提取的贝叶斯最优充分统计量,直到某些单调变换,其性能与理想 oracle 估计器在非线性领域相当。利用这种分析真实值,通过 logit 镜和电路对齐进行机制分析,表明模型不依赖于固定的核平滑启发式方法。相反,它似乎适应了决策变为线性可解的点:在线性任务中表现出一致的投票式集合模式,在非线性任务中则利用更深层的顺序计算。这些发现表明,ICL源于任务自适应的统计估计器的构建,而非简单的相似性匹配。

英文摘要:

In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates, yet the underlying algorithms remain poorly understood. We adopt a statistical decision-theoretic perspective by investigating simple binary hypothesis testing, where the optimal policy is determined by the likelihood-ratio test. Notably, this setup provides a mathematically rigorous setting for mechanistic interpretability where the target algorithmic ground truth is known. By training Transformers on tasks requiring distinct geometries (linear shifted means vs. nonlinear variance estimation), we demonstrate that the models approximate the Bayes-optimal sufficient statistics from context up to some monotonic transformation, matching the performance of an ideal oracle estimator in nonlinear regimes. Leveraging this analytical ground truth, mechanistic analysis via logit lens and circuit alignment suggests that the model does not rely on a fixed kernel smoothing heuristic. Instead, it appears to adapt the point at which decisions become linearly decodable: exhibiting patterns consistent with a voting-style ensemble for linear tasks while utilizing a deeper sequential computation for nonlinear tasks. These findings suggest that ICL emerges from the construction of task-adaptive statistical estimators rather than simple similarity matching.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑