突破相关性平台期:关于基于注意力的回归器优化与容量限制的研究
Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors
AI总结:
本文提出ECA模型,通过理论驱动的机制突破基于注意力的回归器中PCC平台期,提升相关性的同时保持MSE性能。
摘要:
基于注意力的回归模型通常通过联合优化均方误差(MSE)损失和皮尔逊相关系数(PCC)损失来训练,分别强调误差的大小和目标的顺序或形状。训练过程中常见但理解不足的现象是PCC平台期:PCC在训练早期停止提高,即使MSE继续下降。我们提供了这一行为的第一个严格理论分析,揭示了优化动态和模型容量方面的根本限制。首先,针对PCC曲线的平坦化现象,我们发现了一个关键矛盾:降低MSE(大小匹配)可能反而抑制PCC梯度(形状匹配)。这一问题在softmax注意力机制下尤为严重,特别是当需要聚合的数据高度同质化时。其次,我们识别了模型容量的限制:我们推导出任何凸聚合器(包括softmax注意力)的PCC提升极限,表明输入的凸包严格限制了可实现的PCC增益。我们证明数据同质化加剧了这两种限制。受这些见解启发,我们提出了Extrapolative Correlation Attention(ECA),它结合了新的、理论驱动的机制,以改进PCC优化并超越凸包。在多样化的基准测试中,包括具有挑战性的同质数据设置,ECA一致突破PCC平台期,实现了显著的相关性提升,而不会损害MSE性能。
英文摘要:
Attention-based regression models are often trained by jointly optimizing Mean Squared Error (MSE) loss and Pearson correlation coefficient (PCC) loss, emphasizing the magnitude of errors and the order or shape of targets, respectively. A common but poorly understood phenomenon during training is the PCC plateau: PCC stops improving early in training, even as MSE continues to decrease. We provide the first rigorous theoretical analysis of this behavior, revealing fundamental limitations in both optimization dynamics and model capacity. First, in regard to the flattened PCC curve, we uncover a critical conflict where lowering MSE (magnitude matching) can paradoxically suppress the PCC gradient (shape matching). This issue is exacerbated by the softmax attention mechanism, particularly when the data to be aggregated is highly homogeneous. Second, we identify a limitation in the model capacity: we derived a PCC improvement limit for any convex aggregator (including the softmax attention), showing that the convex hull of the inputs strictly bounds the achievable PCC gain. We demonstrate that data homogeneity intensifies both limitations. Motivated by these insights, we propose the Extrapolative Correlation Attention (ECA), which incorporates novel, theoretically-motivated mechanisms to improve the PCC optimization and extrapolate beyond the convex hull. Across diverse benchmarks, including challenging homogeneous data setting, ECA consistently breaks the PCC plateau, achieving significant improvements in correlation without compromising MSE performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。