SIFT-50M：用于语音指令微调的大规模多语言数据集
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
AI总结：
该研究构建含5000万样本的多语言语音指令微调数据集SIFT-50M，训练的SIFT-LLM在指令遵循基准上优于现有模型，同时提出EvalSIFT基准支持相关研究。
摘要：
我们引入SIFT（语音指令微调，Speech Instruction Fine-Tuning），这是一个含5000万样本的数据集，专为语音文本大型语言模型（LLMs）的指令微调与预训练设计。SIFT-50M基于公开语音语料库构建，共包含1.4万小时语音，利用LLMs及现成专家模型；覆盖5种语言，包含多样的语音理解及可控语音生成指令。通过SIFT-50M训练的SIFT-LLM在指令遵循基准上优于现有语音文本LLMs，且在基础语音任务上性能具竞争力；我们还引入EvalSIFT基准，用于评估语音文本LLMs的指令遵循能力。
英文摘要：
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
