发表机构
LY Corporation（LY Corporation）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究证明，大规模对抗预训练的线性Transformer可通过上下文学习将鲁棒性迁移至未见的高斯混合分类任务，渐近达到最优鲁棒贝叶斯误差，而标准训练模型无法实现。
AI 中文摘要
对抗训练是针对对抗攻击最可靠的防御手段之一，但其高昂的计算成本通常需要针对每个任务重新支付。鲁棒基础模型提供了一种有前景的替代方案：对模型进行一次对抗预训练，然后通过轻量级适配将其鲁棒性迁移到下游任务。然而，一个基本问题仍然悬而未决：在预训练期间获得的鲁棒性能否在无需进一步对抗训练的情况下迁移到未见过的任务？在本研究中，我们对此问题给出肯定回答。一个在大规模数据上经过对抗预训练的单一模型，可以在新任务上实现最优鲁棒性，而无需额外的任务特定训练。具体而言，我们证明，对于一类高斯混合分类任务，一个足够深的线性Transformer在跨任务对抗训练后，可以通过从干净示范中进行上下文学习，在未见过的任务上渐近地达到鲁棒贝叶斯误差。相比之下，标准训练的模型则无法做到这一点。我们进一步分析了梯度流下的收敛性、准确率-鲁棒性权衡以及示范复杂度。
英文摘要
Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task. Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream tasks through lightweight adaptation. However, a fundamental question remains open: can robustness acquired during pretraining transfer to unseen tasks without further adversarial training? In this study, we answer this question affirmatively. A single model adversarially pretrained at scale can achieve optimal robustness on new tasks without additional task-specific training. Specifically, we show that, for a family of Gaussian-mixture classification tasks, a sufficiently deep linear transformer adversarially trained across tasks can asymptotically attain the robust Bayes error on previously unseen tasks through in-context learning from clean demonstrations. By contrast, a standardly trained model cannot. We further analyze convergence under gradient flow, an accuracy--robustness trade-off, and demonstration complexity.
