arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.14850cs.CLcs.AI
收藏

关于Long Range Arena中的局部性偏差与结果

On the locality bias and results in the Long Range Arena

Technical University of Madrid(马德里理工大学)

Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho

AI总结:

本文揭示LRA基准实际主要衡量短程依赖,指出Transformer与SSM等架构的性能差异源于局部性偏差,并提出通过训练技巧和位置编码提升Transformer至最优,同时建议重新设计基准。

摘要:

Long Range Arena(LRA)基准旨在评估Transformer改进模型及其替代方案在长程依赖建模任务中的性能。Transformer及其主要变体在此基准上表现不佳,而一系列新架构,如状态空间模型(SSMs),获得了一定关注,在LRA中大幅超越了Transformer。近期研究表明,通过去噪预训练阶段,Transformer可以在LRA中取得与这些新架构相竞争的结果。在本工作中,我们讨论并解释了MEGA和SSMs等架构在Long Range Arena中的优越性,以及近期Transformer结果的改进,指出任务的位置性和局部性本质。我们表明,尽管LRA是用于长程依赖建模的基准,但实际上大部分性能来自于短程依赖。通过使用训练技术来缓解数据效率低下问题,Transformer在适当的 positional encoding(位置编码)下能够达到最先进的性能。此外,采用相同技术,我们能够移除SSM卷积核的所有限制,并学习完全参数化的卷积,而性能不降,这表明SSM背后的设计选择仅为这些特定任务增加了归纳偏置和学习效率。我们的见解表明,LRA结果应谨慎解读,并呼吁重新设计该基准。

英文摘要:

The Long Range Arena (LRA) benchmark was designed to evaluate the performance of Transformer improvements and alternatives in long-range dependency modeling tasks. The Transformer and its main variants performed poorly on this benchmark, and a new series of architectures such as State Space Models (SSMs) gained some traction, greatly outperforming Transformers in the LRA. Recent work has shown that with a denoising pre-training phase, Transformers can achieve competitive results in the LRA with these new architectures. In this work, we discuss and explain the superiority of architectures such as MEGA and SSMs in the Long Range Arena, as well as the recent improvement in the results of Transformers, pointing to the positional and local nature of the tasks. We show that while the LRA is a benchmark for long-range dependency modeling, in reality most of the performance comes from short-range dependencies. Using training techniques to mitigate data inefficiency, Transformers are able to reach state-of-the-art performance with proper positional encoding. In addition, with the same techniques, we were able to remove all restrictions from SSM convolutional kernels and learn fully parameterized convolutions without decreasing performance, suggesting that the design choices behind SSMs simply added inductive biases and learning efficiency for these particular tasks. Our insights indicate that LRA results should be interpreted with caution and call for a redesign of the benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑