arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.25409cs.CLcs.AI
收藏

BhashaBench V1:面向印度领域象限的综合基准

BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains

BharatGen Team(BharatGen团队)
Vijay Devane, Mohd Nauman, Bhargav Patel, Aniket Mahendra Wakchoure, Yogeshkumar Sant, Shyam Pawar, Viraj Thakur, Ananya Godse, Sunil Patra, Neha Maurya, Suraj … 展开作者

Vijay Devane, Mohd Nauman, Bhargav Patel, Aniket Mahendra Wakchoure, Yogeshkumar Sant, Shyam Pawar, Viraj Thakur, Ananya Godse, Sunil Patra, Neha Maurya, Suraj Racha, Nitish Kamal Singh, Ajay Nagpal, Piyush Sawarkar, Kundeshwar Vijayrao Pundalik, Rohit Saluja, Ganesh Ramakrishnan

AI总结:

针对现有大模型评估基准以英语为中心、不区分领域的问题,研究推出覆盖印度四大领域的双语基准BhashaBench V1,评估发现模型在低资源领域及印地语内容上表现偏弱,为印度本土领域的大模型评估提供支撑。

摘要:

大语言模型(LLMs)的快速发展使得针对特定领域与文化的评估需求愈发迫切。现有基准大多以英语为中心且不区分领域,限制了其在印度本土语境下的适用性。为填补这一空白,我们推出BhashaBench V1,这是首个聚焦核心印度知识体系的领域专属、多任务双语基准。BhashaBench V1包含74166对精心整理的问答对,其中52494对为英文,21672对为印地语,均源自官方政府考试与领域专业考试。该基准覆盖农业、法律、金融、阿育吠陀四大核心领域,包含90多个子领域、500多个主题,可实现细粒度评估。对29余款大语言模型的评估显示,模型在不同领域和语言上存在显著的性能差距,低资源领域的差距尤为突出。例如,GPT-4o在法律领域的整体准确率达76.49%,但在阿育吠陀领域仅为59.74%。在所有领域中,模型在英文内容上的表现始终优于印地语内容。子领域层面的分析显示,网络法、国际金融等领域表现相对较好,而五业疗法(Panchakarma)、种子科学、人权等领域的表现明显偏弱。BhashaBench V1为评估大语言模型在印度多元知识领域的表现提供了综合数据集,可用于评估模型将领域专属知识与双语理解能力相结合的水平。所有代码、基准及资源均公开可用,以支持开放研究。

英文摘要:

The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑