CCFQA：跨语言和跨模态语音与文本事实性评估基准
CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
AI总结：
CCFQA是一个用于评估多模态大语言模型跨语言和跨模态事实性能力的基准，通过少样本迁移学习策略在多语言语音问答任务中取得与GPT-4o-mini-Audio相当的性能。
摘要：
随着大型语言模型（LLMs）在多语言世界中的日益普及，确保无幻觉的事实性变得尤为关键。然而，现有的评估多模态大型语言模型（MLLMs）可靠性的基准主要侧重于文本或视觉模态，且以英语为主，这在处理多语言输入，尤其是语音时存在评估缺口。为此，我们提出了一种新的跨语言和跨模态事实性基准（CCFQA）。具体而言，CCFQA基准包含8种语言的平行语音-文本事实性问题，旨在系统评估MLLMs的跨语言和跨模态事实性能力。我们的实验结果表明，当前的MLLMs在CCFQA基准上仍面临显著挑战。此外，我们提出了一种少样本迁移学习策略，能够有效地将LLMs在英语中的问答能力转移到多语言语音问答（SQA）任务中，仅使用5个样本训练即可达到与GPT-4o-mini-Audio相当的性能。我们发布CCFQA作为基础研究资源，以促进具有更稳健和可靠语音理解能力的MLLMs的发展。我们的代码和数据集可在https://github.com/yxduir/ccfqa获取。
英文摘要：
As Large Language Models (LLMs) are increasingly popularized in the multilingual world, ensuring hallucination-free factuality becomes markedly crucial. However, existing benchmarks for evaluating the reliability of Multimodal Large Language Models (MLLMs) predominantly focus on textual or visual modalities with a primary emphasis on English, which creates a gap in evaluation when processing multilingual input, especially in speech. To bridge this gap, we propose a novel Cross-lingual and Cross-modal Factuality benchmark (CCFQA). Specifically, the CCFQA benchmark contains parallel speech-text factual questions across 8 languages, designed to systematically evaluate MLLMs' cross-lingual and cross-modal factuality capabilities. Our experimental results demonstrate that current MLLMs still face substantial challenges on the CCFQA benchmark. Furthermore, we propose a few-shot transfer learning strategy that effectively transfers the Question Answering (QA) capabilities of LLMs in English to multilingual Spoken Question Answering (SQA) tasks, achieving competitive performance with GPT-4o-mini-Audio using just 5-shot training. We release CCFQA as a foundational research resource to promote the development of MLLMs with more robust and reliable speech understanding capabilities. Our code and dataset are available at https://github.com/yxduir/ccfqa.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
