ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models
Nanyang Technological University(南洋理工大学);Independent Researcher(独立研究者);United Arab Emirates University(阿联酋大学);North China Electric Power University(华北电力大学);Southwest Jiaotong University(西南交通大学);Squirrel AI Learning(Squirrel AI学习)
Although Audio Large Language Models (ALLMs) have witnessed substantial advancements, their long audio understanding capabilities remain unexplored. A plethora of benchmarks have been proposed for general audio tasks, they predominantly focus on short-form clips, leaving without a consensus on evaluating ALLMs over extended durations. This paper proposes ChronosAudio, the first multi-task benchmark tailored for long-audio understanding in ALLMs. It encompasses six major task categories and comprises 36,000 test instances totaling over 200 hours audio, stratified into short, middle, and long-form categories to comprehensively evaluate length generalization. Extensive experiments on 16 state-of-the-art models using ChronosAudio yield three critical findings: 1.Precipitous Long-Context Collapse: ALLMs exhibit a severe inability to sustain performance, with the transition from short to long contexts triggering a staggering performance degradation of over 90% in specific tasks. 2.Structural Attention Dilution: Performance degradation stems from a fundamental failure in maintaining temporal locality; attention mechanisms suffer from significant diffusion in later sequences. 3.Restorative Ceiling of Mitigation: Current strategies only offer 50% recovery. These findings reveal significant challenges in long-audio, underscoring the urgent need for approaches to achieve robust, document-level audio reasoning.
Hearing is Believing? Evaluating and Analyzing Audio Language Model Sycophancy with SYAUDIO
University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed人工智能大学);King Abdullah University of Science and Technology(卡布斯大学);Georgia Institute of Technology(佐治亚理工学院)
Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Imperial College London(帝国理工学院);Technische Universität München(慕尼黑工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Shanghai Jiao Tong University(上海交通大学);Johns Hopkins University(约翰斯·霍普金斯大学)
Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models
School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);Singapore University of Technology and Design(新加坡科技设计大学);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦大学玛丽女王学院电子工程与计算机科学学院);School of Statistics and Data Science, Nankai University(南开大学统计与数据科学学院)
VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(台湾大学通信工程研究所);NVIDIA, Taiwan(台湾NVIDIA);Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan(台湾大学人工智能卓越研究中心)
CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Jiao Tong University(上海交通大学);IEIT Systems Co Ltd(IEIT系统有限公司)
Evaluating robustness of You Only Hear Once(YOHO) Algorithm on noisy audios in the VOICe Dataset
Manipal Institute of Technology(曼尼帕尔理工学院)
机器监听中的声音事件检测(SED)需要识别音频文件中的不同声音并识别音频中特定声音事件的开始和结束时间。SED可用于各种应用,例如音频监视、语音识别以及多媒体数据库中的数据的基于上下文的索引和检索。然而,在现实生活中,来自各种来源的音频很少没有任何干扰噪声或干扰。在本文中,我们测试的性能,你只听到一次(YOHO)算法对嘈杂的音频数据。受计算机视觉中You Only Look Once(YOLO)算法的启发,YOHO算法可以在音乐语音检测数据集、TUT声音事件和Urban-SED数据集等数据集上与各种最先进算法的性能相匹配,但推理时间更短。在本文中,我们探讨了YOHO算法的VOICe数据集上的性能,包含在不同的声音噪声比(SNR)的噪声的音频文件。YOHO可以超越或至少匹配VOICe数据集论文中报告的最佳性能SED算法,并在更短的时间内进行推断。
Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning
Huawei Technologies, Munich Research Center(华为技术有限公司慕尼黑研究中心);Music Technology Group, Universitat Pompeu Fabra(庞培法布拉大学音乐技术研究组);Audio Research Group, Tampere University(坦佩雷大学音频研究组)
Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions
Department of Intelligent Software, Sungkyunkwan University(智能软件系,成均馆大学);Department of Computer Science and Engineering, Sungkyunkwan University(计算机科学与工程系,成均馆大学)