上下文内通用性就足够了吗？MLP 在上下文内同样具有通用性
Is In-Context Universality Enough? MLPs are Also Universal In-Context
AI总结：
本文通过证明具有可训练激活函数的 MLP 同样具备上下文内通用性，否定了上下文内通用性是 Transformer 相对经典模型优势来源的观点，指出其成功更可能源于归纳偏置或训练稳定性等因素。
摘要：
Transformer 的成功通常与其执行上下文内学习的能力相关联。近期研究表明，Transformer 在上下文内具有通用性，能够逼近上下文（即定义在 $\mathcal{X}\subseteq \mathbb{R}^d$ 上的概率测度）和查询点 $x\in \mathcal{X}$ 的任意实值连续函数。这引出一个问题：上下文内通用性是否解释了它们相对于经典模型的优势？我们通过证明具有可训练激活函数的 MLP 在上下文内同样具有通用性，对此给出否定回答。这表明 Transformer 的成功可能源于其他因素，例如归纳偏置或训练稳定性。
英文摘要：
The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over $\mathcal{X}\subseteq \mathbb{R}^d$) and a query $x\in \mathcal{X}$. This raises the question: Does in-context universality explain their advantage over classical models? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context. This suggests the transformer's success is likely due to other factors like inductive bias or training stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
