arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.07902cs.CLcs.AI
收藏

透过核心能力的视角:大语言模型评估综述

Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

Research Center for Social Computing and Information Retrieval(社会计算与信息检索研究中心);Harbin Institute of Technology(哈尔滨工业大学)

Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, Ting Liu

AI总结:

本文综述LLM评估研究,提出按推理、知识、可靠性与安全四大核心能力组织基准与指标,并展望未来评估方向。

摘要:

从预训练语言模型(PLM)到大语言模型(LLM),自然语言处理(NLP)领域见证了性能的急剧提升和广泛的实际应用。一个研究领域的评估指引着其改进方向。然而,LLM极难进行全面评估,原因有二:首先,由于LLM的出色表现,传统NLP任务已显得不足;其次,现有评估任务难以跟上现实场景中范围广泛的应用。为解决这些问题,已有工作提出了各种基准以更好地评估LLM。为厘清学术界和工业界众多的评估任务,我们调研了多篇有关LLM评估的论文。我们总结了LLM的4项核心能力,包括推理、知识、可靠性和安全。对于每项能力,我们介绍其定义、相应基准和指标。在这一能力架构下,相似任务被组合起来以反映相应能力,同时新任务也可以被轻松加入该体系。最后,我们就LLM评估的未来方向给出建议。

英文摘要:

From pre-trained language model (PLM) to large language model (LLM), the field of natural language processing (NLP) has witnessed steep performance gains and wide practical uses. The evaluation of a research field guides its direction of improvement. However, LLMs are extremely hard to thoroughly evaluate for two reasons. First of all, traditional NLP tasks become inadequate due to the excellent performance of LLM. Secondly, existing evaluation tasks are difficult to keep up with the wide range of applications in real-world scenarios. To tackle these problems, existing works proposed various benchmarks to better evaluate LLMs. To clarify the numerous evaluation tasks in both academia and industry, we investigate multiple papers concerning LLM evaluations. We summarize 4 core competencies of LLM, including reasoning, knowledge, reliability, and safety. For every competency, we introduce its definition, corresponding benchmarks, and metrics. Under this competency architecture, similar tasks are combined to reflect corresponding ability, while new tasks can also be easily added into the system. Finally, we give our suggestions on the future direction of LLM's evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑