arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07910cs.AIcs.CLcs.LG
收藏

PromptBench:用于评估大型语言模型的统一库

PromptBench: A Unified Library for Evaluation of Large Language Models

Microsoft Research Asia(微软亚洲研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology of China(中国科学技术大学);Carnegie Mellon University(卡内基梅隆大学)

Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, Xing Xie

AI总结:

本文提出PromptBench,一个统一、开放且灵活的库,用于评估大型语言模型,包含提示构建、对抗攻击等组件,支持新基准和评估协议研究。

摘要:

大型语言模型(LLM)的评估对于评估其性能和缓解潜在安全风险至关重要。在本文中,我们介绍了PromptBench,一个用于评估LLM的统一库。它包含几个易于研究人员使用和扩展的关键组件:提示构建、提示工程、数据集和模型加载、对抗性提示攻击、动态评估协议和分析工具。PromptBench旨在成为一个开放、通用且灵活的研究用代码库,可促进在创建新基准、部署下游应用和设计新评估协议方面的原创研究。代码可在https://github.com/microsoft/promptbench获取,并将持续得到支持。

英文摘要:

The evaluation of large language models (LLMs) is crucial to assess their performance and mitigate potential security risks. In this paper, we introduce PromptBench, a unified library to evaluate LLMs. It consists of several key components that are easily used and extended by researchers: prompt construction, prompt engineering, dataset and model loading, adversarial prompt attack, dynamic evaluation protocols, and analysis tools. PromptBench is designed to be an open, general, and flexible codebase for research purposes that can facilitate original study in creating new benchmarks, deploying downstream applications, and designing new evaluation protocols. The code is available at: https://github.com/microsoft/promptbench and will be continuously supported.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑