AgriEval:面向大语言模型的综合性中文农业基准测试集
AgriEval: A Comprehensive Chinese Agricultural Benchmark for Large Language Models
AI总结:
针对农业领域大语言模型缺乏评估基准的问题,提出首个综合性中文农业基准AgriEval,覆盖多类别多题型,实验显示多数现有模型准确率不足60%,并探究了性能影响因素与提升策略。
摘要:
在农业领域,大语言模型(LLM)的部署因训练数据和评估基准的匮乏而受阻。为缓解这一问题,我们提出了AgriEval——首个综合性中文农业基准测试集,它具备三大核心特征:(1)全面的能力评估。AgriEval涵盖农业领域6大主类别和29个子类别,覆盖记忆、理解、推理、生成四大核心认知场景。(2)高质量数据。该数据集从大学阶段的考试与作业中整理而来,为评估大语言模型的知识应用能力及类专家决策能力提供了自然且可靠的基准。(3)格式多样、规模庞大。AgriEval包含14697道选择题和2167道开放式问答题,是目前现有规模最大的农业基准测试集。我们还对51个开源及商用大语言模型开展了全面实验。实验结果显示,现有大多数大语言模型的准确率难以达到60%,凸显了农业大语言模型的发展潜力。此外,我们通过大量实验探究了影响模型性能的因素,并提出了性能提升策略。AgriEval可通过https://github.com/YanPioneer/AgriEval/ 获取。
英文摘要:
In the agricultural domain, the deployment of large language models (LLMs) is hindered by the lack of training data and evaluation benchmarks. To mitigate this issue, we propose AgriEval, the first comprehensive Chinese agricultural benchmark with three main characteristics: (1) Comprehensive Capability Evaluation. AgriEval covers six major agriculture categories and 29 subcategories within agriculture, addressing four core cognitive scenarios: memorization, understanding, inference, and generation. (2) High-Quality Data. The dataset is curated from university-level examinations and assignments, providing a natural and robust benchmark for assessing the capacity of LLMs to apply knowledge and make expert-like decisions. (3) Diverse Formats and Extensive Scale. AgriEval comprises 14,697 multiple-choice questions and 2,167 open-ended question-and-answer questions, establishing it as the most extensive agricultural benchmark available to date. We also present comprehensive experimental results over 51 open-source and commercial LLMs. The experimental results reveal that most existing LLMs struggle to achieve 60% accuracy, underscoring the developmental potential in agricultural LLMs. Additionally, we conduct extensive experiments to investigate factors influencing model performance and propose strategies for enhancement. AgriEval is available at https://github.com/YanPioneer/AgriEval/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。