arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.17628cs.LG
收藏

平滑性是鲁棒性的关键吗?使用一种新度量比较注意力模型与卷积模型

Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric

The University of Tokyo(东京大学)

Baiyuan Chen

AI总结:

本文提出基于逐层拓扑与Lipschitz连续性分析的TopoLip度量,统一比较不同架构的鲁棒性,并证明注意力模型通常比卷积模型更平滑且更鲁棒。

摘要:

鲁棒性是机器学习模型的一个关键方面。现有的鲁棒性评估方法往往缺乏理论通用性,或严重依赖经验性评估,这限制了对促成鲁棒性的结构性因素的理解。此外,理论鲁棒性分析无法直接用于模型之间的比较。为应对这些挑战,我们提出了 TopoLip,这是一种基于逐层分析的度量,它将拓扑数据分析与 Lipschitz 连续性联系起来,用于鲁棒性评估。TopoLip 为不同架构或配置之间的理论和经验鲁棒性比较提供了一个统一框架,并揭示了模型参数如何影响模型的鲁棒性。利用 TopoLip,我们证明基于注意力的模型通常表现出比基于卷积的模型更平滑的变换和更强的鲁棒性,这一点通过理论分析和对抗性任务得到了验证。我们的研究结果建立了架构设计、鲁棒性和拓扑性质之间的联系。

英文摘要:

Robustness is a critical aspect of machine learning models. Existing robustness evaluation approaches often lack theoretical generality or rely heavily on empirical assessments, limiting insights into the structural factors contributing to robustness. Moreover, theoretical robustness analysis is not applicable for direct comparisons between models. To address these challenges, we propose $\textit{TopoLip}$, a metric based on layer-wise analysis that bridges topological data analysis and Lipschitz continuity for robustness evaluation. TopoLip provides a unified framework for both theoretical and empirical robustness comparisons across different architectures or configurations, and it reveals how model parameters influence the robustness of models. Using TopoLip, we demonstrate that attention-based models typically exhibit smoother transformations and greater robustness compared to convolution-based models, as validated through theoretical analysis and adversarial tasks. Our findings establish a connection between architectural design, robustness, and topological properties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑