Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.
Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification
Bias Variance Labs, d.o.o.（Bias Variance Labs有限责任公司）；Faculty of Computer Science and Engineering, University Ss Cyril and Methodius（圣西里尔与美多德大学计算机科学与工程学院）；Jožef Stefan Institute（约泽夫·斯蒂芬研究所）；University of Cambridge（剑桥大学）
VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Zhongguancun Academy（中关村学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；PKU-PsiBot Joint Lab（北京大学-PsiBot联合实验室）
Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting
Karlsruhe Institute of Technology, Institute for Industrial Production（卡尔斯鲁厄理工学院，工业生产研究所）；University of Duisburg-Essen, House of Energy Markets and Finance（杜伊斯堡-埃森大学，能源市场与金融中心）；Karlsruhe Institute of Technology, Institute for Automation and Applied Informatics（卡尔斯鲁厄理工学院，自动化与应用信息研究所）；Karlsruhe Institute of Technology, Institute of Statistics（卡尔斯鲁厄理工学院，统计研究所）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；Marburg University, Department of Mathematics and Computer Science（马尔堡大学，数学与计算机科学系）；Volatile GmbH（Volatile GmbH公司）；Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems – Jülich Systems Analysis（焦耳研究中心 GmbH，气候与能源系统研究所——焦耳系统分析）
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
Salesforce AI Research（Salesforce AI研究院）；Hong Kong Baptist University（香港浸会大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；The Australian National University（澳大利亚国立大学）
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Shenzhen Loop Area Institute（深圳河套学院）；CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学CentraleSupélec分校）；National University of Singapore（新加坡国立大学）
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Software College, Northeastern University（东北大学软件学院）；Shenzhen University of Advanced Technology（深圳大学先进技术学院）；Shenzhen Key Laboratory for High Performance Data Mining（深圳高性能数据挖掘重点实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；University of California, Irvine（加州大学 Irvine 分校）；School of Biomedical Engineering, UNSW Sydney（悉尼大学生物医学工程学院）
Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability
miHoYo Honkai AI R&D Team（米哈游崩坏AI研发团队）；College AI, Tsinghua University（清华大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；MMLab, The University of Hong Kong（香港大学多媒体实验室）；National University of Singapore（新加坡国立大学）
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
MATS；Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Astra Fellowship；ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center；LawZero；Google DeepMind（谷歌DeepMind）
Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models
Tallinn University of Technology（塔林技术大学）；MBZUAI（穆罕默德·本·拉希德·阿勒马克图姆智能研究院）；Idiap Research Institute（伊迪普研究 institute）；Univ. Rennes, CNRS, Irisa（里昂大学，CNRS，Irisa）；Univ. Bretagne Sud, CNRS, Irisa（布列塔尼南部大学，CNRS，Irisa）；Validsoft Ltd.（Validsoft有限公司）
Cohere Labs（Cohere 实验室）；Cohere（Cohere 公司）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；University of Waterloo（滑铁卢大学）；Massachusetts Institute of Technology（麻省理工学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Om AI Research（Om AI研究院）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Edinburgh（爱丁堡大学）
Open-Source Manually Annotated Vocal Tract Database for Automatic Segmentation from 3D MRI Using Deep Learning: Benchmarking 2D and 3D Convolutional and Transformer Networks
University of Iowa（爱荷华大学）；Shenandoah University（谢南多厄大学）
3DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models
Fudan University（复旦大学）；Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Stanford University（斯坦福大学）；The Chinese University of Hong Kong（香港中文大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
