BhashaBench V1:面向印度领域象限的综合基准
BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains
AI总结:
针对现有大模型评估基准以英语为中心、不区分领域的问题,研究推出覆盖印度四大领域的双语基准BhashaBench V1,评估发现模型在低资源领域及印地语内容上表现偏弱,为印度本土领域的大模型评估提供支撑。
摘要:
大语言模型(LLMs)的快速发展使得针对特定领域与文化的评估需求愈发迫切。现有基准大多以英语为中心且不区分领域,限制了其在印度本土语境下的适用性。为填补这一空白,我们推出BhashaBench V1,这是首个聚焦核心印度知识体系的领域专属、多任务双语基准。BhashaBench V1包含74166对精心整理的问答对,其中52494对为英文,21672对为印地语,均源自官方政府考试与领域专业考试。该基准覆盖农业、法律、金融、阿育吠陀四大核心领域,包含90多个子领域、500多个主题,可实现细粒度评估。对29余款大语言模型的评估显示,模型在不同领域和语言上存在显著的性能差距,低资源领域的差距尤为突出。例如,GPT-4o在法律领域的整体准确率达76.49%,但在阿育吠陀领域仅为59.74%。在所有领域中,模型在英文内容上的表现始终优于印地语内容。子领域层面的分析显示,网络法、国际金融等领域表现相对较好,而五业疗法(Panchakarma)、种子科学、人权等领域的表现明显偏弱。BhashaBench V1为评估大语言模型在印度多元知识领域的表现提供了综合数据集,可用于评估模型将领域专属知识与双语理解能力相结合的水平。所有代码、基准及资源均公开可用,以支持开放研究。
英文摘要:
The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。