VākQA：泰卢固语口语事实型问答的基准与评估研究
VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
浏览论文内容
中文总结 AI 辅助
本文提出泰卢固语口语问答基准VākQA，含2,001个问答对，验证评估方法并基准测试模型，发现翻译丢失文化特异性、语音混淆及级联错误等问题。
中文摘要 AI 辅助
随着大语言模型的发展，问答技术取得了快速进步，但主要集中于高资源语言，且涵盖文本和口语两种场景。泰卢固语的口语问答（SQA）基准尚未被探索，且该场景下自动评估的可靠性也未被量化。我们引入了VākQA，一个泰卢固语SQA基准，包含六个领域的2,001个事实型问答对，配有2.53小时的语音音频、双语转写以及人工验证的参考答案。我们首先对照人工评判验证了评估方法：Gemini作为评判者最接近人工评分，但存在非均匀的严格性，而开放权重模型作为评判者则系统性地惩罚了与参考表面形式不同的正确泰卢固语答案。利用这一经过验证的设置，我们对专有和开放权重模型在输入模态、语言和领域方面进行了基准测试。我们观察到，泰卢固语的措辞保留了翻译中会丢失的文化特异性，语音输入会引入改变问题含义的语音混淆，而级联的ASR-MT错误会逐步累积。VākQA已公开发布。
英文摘要
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.
发表机构
- International Institute of Information Technology Hyderabad（海得拉巴国际信息技术学院）
- Brno University of Technology（布尔诺理工大学）
机构由 AI 辅助整理，请以论文原文为准。
