arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10950cs.CL
收藏

E-Bench:迈向评估大型语言模型的易用性

E-Bench: Towards Evaluating the Ease-of-Use of Large Language Models

Baidu Inc.(百度公司);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所)

Zhenyu Zhang, Bingguang Hao, Jinpeng Li, Zekai Zhang, Dongyan Zhao

AI总结:

该研究构建 E-Bench 基准,通过同义、排版及其组合扰动评估大型语言模型在真实提示变化下的易用性,并分析性能下降原因。

摘要:

大多数大型语言模型(LLM)对提示词很敏感,另一种同义表达或一个拼写错误都可能导致模型产生意外结果。为特定需求编写最优提示词缺乏理论支持,完全依赖人工试验,这给生成式人工智能的普及造成了相当大的障碍。然而,目前尚无关于 LLM 在真实场景中抵抗提示词扰动稳定性的系统性分析。在本研究中,我们提出评估 LLM 的易用性,并构建 E-Bench,从同义扰动(包括改写、简化和口语化)与排版扰动(如打字输入)两个方面模拟人类使用的实际情况。在此基础上,我们还讨论了这两类扰动的组合,并分析了性能下降的主要原因。实验结果表明,随着模型规模的增加,尽管易用性显著提升,但要构建足够用户友好的模型仍有很长的路要走。

英文摘要:

Most large language models (LLMs) are sensitive to prompts, and another synonymous expression or a typo may lead to unexpected results for the model. Composing an optimal prompt for a specific demand lacks theoretical support and relies entirely on human experimentation, which poses a considerable obstacle to popularizing generative artificial intelligence. However, there is no systematic analysis of the stability of LLMs in resisting prompt perturbations in real-world scenarios. In this work, we propose to evaluate the ease-of-use of LLMs and construct E-Bench, simulating the actual situation of human use from synonymous perturbation (including paraphrasing, simplification, and colloquialism) and typographical perturbation (such as typing). On this basis, we also discuss the combination of these two types of perturbation and analyze the main reasons for performance degradation. Experimental results indicate that with the increase of model size, although the ease-of-use are significantly improved, there is still a long way to go to build a sufficiently user-friendly model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑