EnviroExam：大型语言模型环境科学知识基准评测
EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models
AI总结：
本文提出 EnviroExam 基准，基于顶尖高校42门课程的936题，对31个开源大模型进行0-shot与5-shot环境科学知识评测，并以变异系数多角度分析模型表现。
摘要：
在环境科学领域，为大型语言模型建立可靠的评估指标以确保其有效性和准确性至关重要。我们提出 EnviroExam，这是一种旨在评估大型语言模型在环境科学领域知识的综合评测方法。EnviroExam 基于国际顶尖高校的课程体系，覆盖本科、硕士和博士课程，包含横跨42门核心课程的936道题目。通过对31个开源大型语言模型进行0-shot和5-shot测试，EnviroExam 揭示了这些模型在环境科学领域的性能差异，并提供了详细的评估标准。结果显示，61.3%的模型通过了5-shot测试，而48.39%的模型通过了0-shot测试。通过引入变异系数作为指标，我们从多个角度评估主流开源大型语言模型在环境科学中的表现，为该领域选择和微调语言模型提供有效标准。未来研究将使用专业环境科学教材构建更多领域专用测试集，以进一步提升评估的准确性和专门性。
英文摘要：
In the field of environmental science, it is crucial to have robust evaluation metrics for large language models to ensure their efficacy and accuracy. We propose EnviroExam, a comprehensive evaluation method designed to assess the knowledge of large language models in the field of environmental science. EnviroExam is based on the curricula of top international universities, covering undergraduate, master's, and doctoral courses, and includes 936 questions across 42 core courses. By conducting 0-shot and 5-shot tests on 31 open-source large language models, EnviroExam reveals the performance differences among these models in the domain of environmental science and provides detailed evaluation standards. The results show that 61.3% of the models passed the 5-shot tests, while 48.39% passed the 0-shot tests. By introducing the coefficient of variation as an indicator, we evaluate the performance of mainstream open-source large language models in environmental science from multiple perspectives, providing effective criteria for selecting and fine-tuning language models in this field. Future research will involve constructing more domain-specific test sets using specialized environmental science textbooks to further enhance the accuracy and specificity of the evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
