arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.02069cs.CLcs.AIcs.LG
收藏

LMentry:一个基本语言任务的语言模型基准

LMentry: A Language Model Benchmark of Elementary Language Tasks

Tel Aviv University(特拉维夫大学);Meta AI

Avia Efrat, Or Honovich, Omer Levy

摘要:

随着大语言模型性能的快速提高,基准测试也变得越来越大、越来越复杂。我们提出了LMentry,一个基准,避免这种“军备竞赛”,专注于一组紧凑的任务,是微不足道的人类,例如写一个句子包含一个特定的词,确定哪些词在列表中属于一个特定的类别,或选择两个词中的哪一个更长。LMentry专门设计用于提供对大语言模型的功能和健壮性的快速和可解释的见解。我们的实验揭示了各种各样的失败案例,虽然对人类来说是显而易见的,但对大语言模型构成了相当大的挑战,包括OpenAI最新的175 B参数优化模型TextDavinci002。LMentry补充了大语言模型的当代评估方法,提供了一个快速,自动和易于运行的“单元测试“,而无需求助于复杂任务的大型基准测试套件。

英文摘要:

As the performance of large language models rapidly improves, benchmarks are getting larger and more complex as well. We present LMentry, a benchmark that avoids this "arms race" by focusing on a compact set of tasks that are trivial to humans, e.g. writing a sentence containing a specific word, identifying which words in a list belong to a specific category, or choosing which of two words is longer. LMentry is specifically designed to provide quick and interpretable insights into the capabilities and robustness of large language models. Our experiments reveal a wide variety of failure cases that, while immediately obvious to humans, pose a considerable challenge for large language models, including OpenAI's latest 175B-parameter instruction-tuned model, TextDavinci002. LMentry complements contemporary evaluation approaches of large language models, providing a quick, automatic, and easy-to-run "unit test", without resorting to large benchmark suites of complex tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑