M3-SLU：评估多模态大语言模型中的说话人归因推理能力
M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models
AI总结：
本研究提出多模态大语言模型基准M3-SLU，用于评估多说话人多轮口语理解中的说话人归因推理能力，基于四个公开语料构建含两项任务的测试集，实验发现现有模型在说话人识别上存在明显缺陷。
摘要：
我们提出了M3-SLU，这是一个全新的多模态大语言模型（MLLM）基准，用于评估多说话人、多轮口语理解能力。尽管近期模型在语音和文本理解上表现出色，但它们在说话人归因推理——即理解自然对话中谁在何时说了什么的能力——方面仍存在困难。M3-SLU基于四个公开语料库（CHiME-6、MELD、MultiDialog和AMI）构建，包含超过12000个经过验证的样本，配有音频、转录文本和元数据。该基准包含两项任务：（1）说话人归因问答，（2）基于话语匹配的说话人归因。我们提供了级联流水线和端到端MLLM的基线结果，采用LLM-as-Judge（大语言模型作为评判器）和准确率指标进行评估。结果显示，尽管模型能够理解对话内容，但往往无法识别说话人身份，暴露了说话人感知对话理解中的关键缺陷。M3-SLU作为一项具有挑战性的基准，将推动说话人感知多模态理解领域的研究。
英文摘要：
We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
