arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.04828cs.CLcs.AIcs.LG
收藏

超越下一词预测:为衡量大语言模型在现实世界应用中的表现开发综合评估框架

Beyond Next Word Prediction: Developing Comprehensive Evaluation Frameworks for measuring LLM performance on real world applications

AMD(超威半导体公司);LayerLens;UCLA(加利福尼亚大学洛杉矶分校)

Vishakha Agrawal, Archie Chaudhury, Shreya Agrawal

AI总结:

本文提出一种基于游戏与工具架构的通用评估框架,以超越静态数据集的方式全面衡量大语言模型在供应链、金融推理及伦理安全等现实场景中的综合能力。

摘要:

尽管大语言模型(LLMs)本质上是下一词元预测系统,但其实际应用远不止这一基本功能。从自然语言处理和文本生成到对话助手和软件使用,LLMs 拥有众多用例,并且已经在企业中获得相当程度的采用。为了评估此类模型,通常使用静态评估数据集(由一组提示及其对应的真实答案组成)来基准测试模型在特定任务上的有效性。在本文中,我们为一个更全面的评估框架提供了基础,该框架基于传统的游戏和工具架构,能够对模型能力进行更全面的衡量。为简单起见,我们提供了一个通用基础,无需重大改动即可扩展到众多场景,从供应链管理或金融推理等具体用例,到伦理或安全等抽象测量。

英文摘要:

While Large Language Models (LLMs) are fundamentally next-token prediction systems, their practical applications extend far beyond this basic function. From natural language processing and text generation to conversational assistants and software use, LLMs have numerous use-cases, and have already acquired a significant degree of enterprise adoption. To evaluate such models, static evaluation datasets, consisting of a set of prompts and their corresponding ground truths, are often used to benchmark the efficacy of the model for a particular task. In this paper, we provide the basis for a more comprehensive evaluation framework, based upon a traditional game and tool-based architecture that enables a more overarching measurement of a model's capabilities. For simplicity, we provide a generalized foundation that can be extended, without significant alteration, to numerous scenarios, from specific use cases such as supply chain management or financial reasoning, to abstract measurements such as ethics or safety.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑