arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01265stat.MLcs.AIcs.LGecon.EMmath.STstat.TH
收藏

Transformer 处理上下文内线性回归中的内生性

Transformers Handle Endogeneity in In-Context Linear Regression

UC Davis(加州大学戴维斯分校)

Haodong Liang, Krishnakumar Balasubramanian, Lifeng Lai

AI总结:

本文研究 transformer 如何利用工具变量处理上下文内线性回归的内生性,揭示其模拟双层优化并指数收敛至 2SLS 的机制,提出预训练方案并通过理论与实验证明其更稳健可靠。

摘要:

我们探究了 transformer 解决上下文内线性回归中内生性的能力。主要发现是,transformer 天然具备一种利用工具变量(IV)有效处理内生性的机制。首先,我们证明 transformer 架构能够模拟一种基于梯度的双层优化过程,该过程以指数速率收敛到广泛使用的两阶段最小二乘(2SLS)解。接着,我们提出一种上下文内预训练方案,并给出理论保证,表明预训练损失的全局最小化点能够取得较小的超额损失。大量实验验证了这些理论发现,表明在存在内生性时,经过训练的 transformer 相比 2SLS 方法能提供更稳健、更可靠的上下文内预测和系数估计。

英文摘要:

We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares $(\textsf{2SLS})$ solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the $\textsf{2SLS}$ method, in the presence of endogeneity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑