Speech-IFEval：评估语音感知语言模型中的指令遵循能力并量化灾难性遗忘
Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models
AI总结：
本文提出Speech-IFEval基准，用于评估语音感知语言模型的指令遵循能力并量化灾难性遗忘，发现多数模型在基本指令上表现远逊于文本模型且对提示变化敏感，强调需超越任务级指标进行评估。
摘要：
我们提出了Speech-IFEval，一个旨在评估语音感知语言模型（SLMs）中指令遵循能力并量化灾难性遗忘的评估框架。最近的SLMs将语音感知与大型语言模型（LLMs）集成，往往由于以语音为中心的训练而降低了文本能力。现有基准混淆了语音感知与指令遵循，阻碍了对这些不同技能的评估。为弥补这一空白，我们提供了一个用于诊断SLMs指令遵循能力的基准。我们的研究结果表明，大多数SLMs即使面对基本指令也表现不佳，其性能远逊于基于文本的LLMs。此外，这些模型对提示变化高度敏感，常常产生不一致且不可靠的输出。我们强调了核心挑战，并为指导未来研究提供了见解，强调需要超越任务级指标的评估。
英文摘要：
We introduce Speech-IFeval, an evaluation framework designed to assess instruction-following capabilities and quantify catastrophic forgetting in speech-aware language models (SLMs). Recent SLMs integrate speech perception with large language models (LLMs), often degrading textual capabilities due to speech-centric training. Existing benchmarks conflate speech perception with instruction-following, hindering evaluation of these distinct skills. To address this gap, we provide a benchmark for diagnosing the instruction-following abilities of SLMs. Our findings show that most SLMs struggle with even basic instructions, performing far worse than text-based LLMs. Additionally, these models are highly sensitive to prompt variations, often yielding inconsistent and unreliable outputs. We highlight core challenges and provide insights to guide future research, emphasizing the need for evaluation beyond task-level metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
