arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09437cs.IRcs.CL
收藏

Hindi-BEIR:印地语大规模检索基准

Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi

Indian Institute of Technology Patna(印度理工学院巴特那分校);IBM Research(IBM研究院)

Arkadeep Acharya, Rudra Murthy, Vishwajeet Kumar, Jaydeep Sen

AI总结:

针对印地语缺乏综合检索评估基准的问题,研究者推出包含15个数据集、覆盖8种任务的Hindi-BEIR基准,并评估前沿多语言检索模型以明确挑战,推动印地语检索研究发展。

摘要:

鉴于全球印地语使用者数量庞大,迫切需要稳健高效的印地语信息检索系统。尽管相关研究仍在推进,但目前缺乏用于评估印地语检索模型的综合基准。为填补这一空白,我们推出了BEIR(异构信息检索基准)的印地语版本,它包含翻译为印地语的英文BEIR数据集子集、已有的印地语检索数据集,以及人工合成的检索数据集。该基准由15个数据集组成,覆盖8种不同任务。我们在该基准上评估了前沿的多语言检索模型,以识别任务和领域特定的挑战及其对检索性能的影响。通过发布该基准和一组相关基线,我们帮助研究者了解当前印地语检索模型的局限性与能力,推动这一关键领域的发展。Hindi-BEIR的数据集已公开可用。

英文摘要:

Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, there is a lack of comprehensive benchmark for evaluating retrieval models in Hindi. To address this gap, we introduce the Hindi version of the BEIR benchmark, which includes a subset of English BEIR datasets translated to Hindi, existing Hindi retrieval datasets, and synthetically created datasets for retrieval. The benchmark is comprised of $15$ datasets spanning across $8$ distinct tasks. We evaluate state-of-the-art multilingual retrieval models on this benchmark to identify task and domain-specific challenges and their impact on retrieval performance. By releasing this benchmark and a set of relevant baselines, we enable researchers to understand the limitations and capabilities of current Hindi retrieval models, promoting advancements in this critical area. The datasets from Hindi-BEIR are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑