arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.13522cs.LG
收藏

INDIC QA BENCHMARK:一个评估LLMs在印度语言中问答能力的多语言基准

INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

Indian Institute of Technology Bombay(印度理工学院孟买分校);IBM Research(IBM研究院)

Abhishek Kumar Singh, Vishwajeet kumar, Rudra Murthy, Jaydeep Sen, Ashish Mittal, Ganesh Ramakrishnan

AI总结:

该论文提出Indic QA Benchmark,一个覆盖11种印度语言的多语言问答基准,发现多语言LLMs因英语偏见在低资源语言中表现弱,而Translate Test范式表现更优。

摘要:

大型语言模型(LLMs)在英语的未见任务上表现良好,但由于基准和训练数据的限制,它们在非英语语言中的能力探索较少。为弥补这一差距,我们引入了Indic QA Benchmark,这是一个涵盖11种主要印度语言的基于上下文问答的大型数据集,包括抽取式和生成式任务。对多语言LLMs(包括指令微调版本)的评估显示,由于训练数据中强烈的英语偏见,它们在低资源语言中表现较弱。我们还研究了Translate Test范式,即输入被翻译成英语进行处理,结果再翻译回源语言输出。这种方法优于多语言LLMs,尤其在低资源环境下。通过发布Indic QA,我们旨在促进对LLMs在低资源语言中问答能力的进一步研究。该基准为解决现有局限和促进多语言理解提供了关键资源。

英文摘要:

Large Language Models (LLMs) perform well on unseen tasks in English, but their abilities in non English languages are less explored due to limited benchmarks and training data. To bridge this gap, we introduce the Indic QA Benchmark, a large dataset for context grounded question answering in 11 major Indian languages, covering both extractive and abstractive tasks. Evaluations of multilingual LLMs, including instruction finetuned versions, revealed weak performance in low resource languages due to a strong English language bias in their training data. We also investigated the Translate Test paradigm,where inputs are translated to English for processing and the results are translated back into the source language for output. This approach outperformed multilingual LLMs, particularly in low resource settings. By releasing Indic QA, we aim to promote further research into LLMs question answering capabilities in low resource languages. This benchmark offers a critical resource to address existing limitations and foster multilingual understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑