AIR-Bench:通过生成式理解评估大型音频-语言模型
AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension
AI总结:
本文提出AIR-Bench,首个评估大型音频-语言模型理解多种音频信号并通过文本与人交互能力的基准,包含基础和聊天两个维度,利用GPT-4评估生成假设,揭示模型局限以指导未来研究。
摘要:
近期,指令遵循的音频-语言模型在人机音频交互方面受到了广泛关注。然而,缺乏能够评估以音频为中心的交互能力的基准,阻碍了该领域的发展。以往的模型主要侧重于评估不同的基础任务,如自动语音识别(ASR),而缺乏对以音频为中心的开放式生成能力的评估。因此,很难追踪大型音频-语言模型(LALMs)领域的进展,并为未来的改进提供指导。本文介绍了AIR-Bench(Audio InstRuction Benchmark),这是首个旨在评估LALMs理解各种类型音频信号(包括人类语音、自然声音和音乐)能力,并进一步以文本格式与人类交互的基准。AIR-Bench包含两个维度:基础基准和聊天基准。前者包含19个任务和约19k个单选题,旨在检验LALMs的基本单任务能力。后者包含2k个开放式问答数据实例,直接评估模型对复杂音频的理解及其遵循指令的能力。两个基准都要求模型直接生成假设。我们设计了一个统一框架,利用先进的语言模型(如GPT-4),根据音频的元信息对生成的假设进行评分。实验结果表明,基于GPT-4的评估与人工评估之间具有高度一致性。通过评估结果揭示现有LALMs的局限性,AIR-Bench能为未来研究方向提供洞察。
英文摘要:
Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio \textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: \textit{foundation} and \textit{chat} benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。