arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.07531cs.CL
收藏

超越基准测试:一种面向大语言模型评估与评定的新范式

Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Models

Jin Liu, Qingquan Li, Wenlong Du

AI总结:

针对现有大语言模型基准存在的内容受限、更新滞后、缺优化指导等问题,提出Benchmarking-Evaluation-Assessment新范式,将评估从“考场”模式转为“体检”模式,可深度归因问题并提供优化建议。

摘要:

当前用于评估大语言模型(LLM)的基准存在评估内容受限、更新不及时、缺乏优化指导等问题。本文提出一种全新的LLM测评范式:Benchmarking-Evaluation-Assessment(基准测试-评估-评定)。该范式将LLM评估的“场地”从“考场”转移至“医院”,通过对LLM开展“体检”,以具体任务求解作为评估内容,对LLM存在的问题进行深度归因,并给出优化建议。

英文摘要:

In current benchmarks for evaluating large language models (LLMs), there are issues such as evaluation content restriction, untimely updates, and lack of optimization guidance. In this paper, we propose a new paradigm for the measurement of LLMs: Benchmarking-Evaluation-Assessment. Our paradigm shifts the "location" of LLM evaluation from the "examination room" to the "hospital". Through conducting a "physical examination" on LLMs, it utilizes specific task-solving as the evaluation content, performs deep attribution of existing problems within LLMs, and provides recommendation for optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑