In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard
Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models
Tallinn University of Technology（塔林技术大学）；MBZUAI（穆罕默德·本·拉希德·阿勒马克图姆智能研究院）；Idiap Research Institute（伊迪普研究 institute）；Univ. Rennes, CNRS, Irisa（里昂大学，CNRS，Irisa）；Univ. Bretagne Sud, CNRS, Irisa（布列塔尼南部大学，CNRS，Irisa）；Validsoft Ltd.（Validsoft有限公司）
Carleton University（卡尔顿大学）；Durham University（杜伦大学）；Stanford University（斯坦福大学）；Aarhus University（奥胡斯大学）；Indian Institute of Technology, Kharagpur（印度理工学院，卡里格普）；Harvard University（哈佛大学）
SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents
DAMO Academy, Alibaba Group（阿里巴巴集团大模型研究院）；Computer Science and Engineering Division, University of Michigan（密歇根大学计算机科学与工程系）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学光明人工智能学院）
