Dynamic-SUPERB:面向语音的动态、协作与全面的指令微调基准
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
AI总结:
本文提出Dynamic-SUPERB,一个面向语音的动态协作指令微调基准,包含55个评估实例,覆盖33个任务和22个数据集,并建立基线模型,发现其在已见任务上表现合理但在未见任务上表现挣扎,以推动通用语音模型发展。
摘要:
文本语言模型在提供良好定义的指令时,展现出在泛化到未见任务上卓越的零样本能力。然而,现有的语音处理研究主要集中在有限或特定任务上。此外,缺乏标准化基准阻碍了不同方法之间的公平比较。因此,我们提出了Dynamic-SUPERB,这是一个旨在构建通用语音模型的基准,能够利用指令微调以零样本方式执行多种任务。为了实现对多样语音任务的全面覆盖并利用指令微调,我们邀请社区协作贡献,促进基准的动态增长。作为启动,Dynamic-SUPERB通过结合33个任务和22个数据集,特色提供了55个评估实例。这跨越了广泛的维度,为评估提供了一个全面的平台。此外,我们提出了几种方法来建立基准基线。这些包括利用语音模型、文本语言模型和多模态编码器。评估结果表明,尽管这些基线在已见任务上表现合理,但在未见任务上表现挣扎。我们向公众发布所有材料,并欢迎研究人员协作参与该项目,共同推进该领域的技术发展。
英文摘要:
Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. However, existing studies in speech processing primarily focus on limited or specific tasks. Moreover, the lack of standardized benchmarks hinders a fair comparison across different approaches. Thus, we present Dynamic-SUPERB, a benchmark designed for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion. To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark. To initiate, Dynamic-SUPERB features 55 evaluation instances by combining 33 tasks and 22 datasets. This spans a broad spectrum of dimensions, providing a comprehensive platform for evaluation. Additionally, we propose several approaches to establish benchmark baselines. These include the utilization of speech models, text language models, and the multimodal encoder. Evaluation results indicate that while these baselines perform reasonably on seen tasks, they struggle with unseen ones. We release all materials to the public and welcome researchers to collaborate on the project, advancing technologies in the field together.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。