标准化陷阱：表格基础模型中的联合标签处理认证

中文摘要 AI 辅助

线性回归和核平滑为上下文学习提供了易于处理的解释：在这两种方法中，特征决定了分配给每个上下文标签的权重。然而，这种固定权重描述是否适用于预训练的表格基础模型（TFMs）仍不清楚。使用导数测试这种描述会陷入标准化陷阱：公开的TFM包在模型看到标签之前对标签进行标准化，但普通导数也反映了标准化标签集之外的行为，这使得模型即使其每个预测都与固定权重映射一致时也显得非线性。我们提出了两种证书，它们仅依赖于标准化标签处的预测，并能拒绝两种不同的解释：固定权重预测和独立非线性标签变换之和。在我们评估的五个公开TFM中，我们的证书表明，改变一个上下文标签会改变其他标签如何影响预测，我们将这种行为称为联合处理。我们进一步发现，联合处理随着训练而出现，并且注意力分数承载了大部分测量的交互作用。总之，这些发现促使TFM解释需考虑上下文标签如何改变单个示例的影响力。

英文摘要

Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label. However, whether this fixed-weight account describes pretrained tabular foundation models (TFMs) remains unclear. Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map. We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations. Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing. We further find that joint processing emerges with training and that attention scores carry most of the measured interaction. Together, these findings motivate TFM explanations that account for how context labels change the influence of individual examples.

发表机构

• Ekimetrics

机构由 AI 辅助整理，请以论文原文为准。