arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17256cs.CL
收藏

超越已知:考察大语言模型在域外意图检测上的性能

Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection

Meituan(美团)

Pei Wang, Keqing He, Yejie Wang, Xiaoshuai Song, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, Weiran Xu

AI总结:

本文系统评估 LLM 在 OOD 意图检测中的表现,发现其零样本和少样本能力较强但不及全资源微调模型,并总结领域知识注入等未来方向。

摘要:

域外(OOD)意图检测旨在考察用户查询是否落在系统预定义域之外,这对任务型对话(TOD)系统的正常运行至关重要。以往方法通过微调判别式模型来解决该问题。近来,一些研究一直在探索以 ChatGPT 为代表的大语言模型(LLM)在各类下游任务中的应用,但其在 OOD 检测任务上的能力仍不清楚。本文在多种实验设置下对 LLM 进行全面评估,并概述 LLM 的优势与弱点。我们发现,LLM 展现出强大的零样本和少样本能力,但与使用完整资源微调的模型相比仍处于劣势。更深入地,通过一系列附加分析实验,我们讨论并总结了 LLM 面临的挑战,并为未来工作提供指导,包括注入领域知识、加强从 IND(域内)到 OOD 的知识迁移,以及理解长指令。

英文摘要:

Out-of-domain (OOD) intent detection aims to examine whether the user's query falls outside the predefined domain of the system, which is crucial for the proper functioning of task-oriented dialogue (TOD) systems. Previous methods address it by fine-tuning discriminative models. Recently, some studies have been exploring the application of large language models (LLMs) represented by ChatGPT to various downstream tasks, but it is still unclear for their ability on OOD detection task.This paper conducts a comprehensive evaluation of LLMs under various experimental settings, and then outline the strengths and weaknesses of LLMs. We find that LLMs exhibit strong zero-shot and few-shot capabilities, but is still at a disadvantage compared to models fine-tuned with full resource. More deeply, through a series of additional analysis experiments, we discuss and summarize the challenges faced by LLMs and provide guidance for future work including injecting domain knowledge, strengthening knowledge transfer from IND(In-domain) to OOD, and understanding long instructions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑