arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12004cs.CL
收藏

理解非常大语言模型对对话评估的有效性

Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation

Carnegie Mellon University(卡内基梅隆大学);Amazon(亚马逊);Microsoft Turing(微软图灵)

Jessica Huynh, Cathy Jiao, Prakhar Gupta, Shikib Mehri, Payal Bajaj, Vishrav Chaudhary, Maxine Eskenazi

摘要:

语言模型的规模在过去几年中稳步增长。它们在各种自然语言处理(NLP)任务(如问答和摘要)上实现了高水平的性能。大语言模型(LLM)已经被用于生成,现在可以输出类似人类的文本。因此,对话领域中的其他下游任务现在可以利用LLM的语言理解能力。对话评估是本文将探讨的任务之一。它专注于LLM的提示:BLOOM,OPT,GPT-3,Flan-T5,InstructDial和TNLGv 2。本文表明,用于训练模型的数据集的选择有助于它在任务上的表现以及提示应该如何构建。具体来说,模型训练的数据集组越多样化和相关,对话评估的性能就越好。本文还研究了提示中的示例数量和使用的示例选择类型如何影响模型的性能。

英文摘要:

Language models have steadily increased in size over the past few years. They achieve a high level of performance on various natural language processing (NLP) tasks such as question answering and summarization. Large language models (LLMs) have been used for generation and can now output human-like text. Due to this, there are other downstream tasks in the realm of dialog that can now harness the LLMs' language understanding capabilities. Dialog evaluation is one task that this paper will explore. It concentrates on prompting with LLMs: BLOOM, OPT, GPT-3, Flan-T5, InstructDial and TNLGv2. The paper shows that the choice of datasets used for training a model contributes to how well it performs on a task as well as on how the prompt should be structured. Specifically, the more diverse and relevant the group of datasets that a model is trained on, the better dialog evaluation performs. This paper also investigates how the number of examples in the prompt and the type of example selection used affect the model's performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑