arXiv 2405.178402024-06-16cs.CL

基准低估了多语言对话智能体的就绪程度

Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents

Stanford University（斯坦福大学）；Université Paris-Saclay（巴黎-萨克雷大学）；CEA（法国原子能和替代能源委员会）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Hanyang University（汉阳大学）；Tianjin University（天津大学）

Andrew H. Lee, Sina J. Semnani, Galo Castillo-López, Gäel de Chalendar, Monojit Choudhury, Ashna Dua, Kapil Rajesh Kavitha, Sungkyun Kim, Prashant Kodali, Ponnu… 展开作者

Andrew H. Lee, Sina J. Semnani, Galo Castillo-López, Gäel de Chalendar, Monojit Choudhury, Ashna Dua, Kapil Rajesh Kavitha, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Alexis Lombard, Mehrad Moradshahi, Gihyun Park, Nasredine Semmar, Jiwon Seo, Tianhao Shen, Manish Shrivastava, Deyi Xiong, Monica S. Lam

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文首次证明上下文学习足以应对多语言任务型对话，通过将对话状态跟踪分解为更简单步骤，并纠正数据集标注错误后，GPT-4在X-RiSAWOZ上取得接近或超过微调SOTA的效果，揭示现有自动指标严重低估了上下文学习能力。

摘要：

由于训练数据获取成本高昂，构建多语言任务型对话（TOD）智能体具有挑战性。遵循提高训练数据效率的研究趋势，我们首次表明，上下文学习足以应对多语言TOD。为了处理具有挑战性的对话状态跟踪（DST）子任务，我们将其分解为更简单的步骤，使其与仅使用少量少样本示例的上下文学习更加兼容。我们在多语言TOD数据集X-RiSAWOZ上测试了我们的方法，该数据集包含中文、英语、法语、韩语、印地语以及印地语-英语代码混合的12个领域。我们在6种语言上的逐轮DST准确率从55.6%到80.3%不等，表面上低于微调模型取得的60.7%到82.8%的SOTA结果；我们在响应生成（RG）子任务中的BLEU分数也显著低于SOTA。然而，在对验证集进行人工评估后，我们发现，通过纠正金标准标签错误并改进数据集标注模式，使用我们提示的GPT-4能够实现（1）DST准确率达到89.6%-96.8%，以及（2）在不同语言上超过99%的正确响应生成。这使我们得出结论：当前的自动评估指标严重低估了上下文学习的有效性。

英文摘要：

Creating multilingual task-oriented dialogue (TOD) agents is challenging due to the high cost of training data acquisition. Following the research trend of improving training data efficiency, we show for the first time, that in-context learning is sufficient to tackle multilingual TOD. To handle the challenging dialogue state tracking (DST) subtask, we break it down to simpler steps that are more compatible with in-context learning where only a handful of few-shot examples are used. We test our approach on the multilingual TOD dataset X-RiSAWOZ, which has 12 domains in Chinese, English, French, Korean, Hindi, and code-mixed Hindi-English. Our turn-by-turn DST accuracy on the 6 languages range from 55.6% to 80.3%, seemingly worse than the SOTA results from fine-tuned models that achieve from 60.7% to 82.8%; our BLEU scores in the response generation (RG) subtask are also significantly lower than SOTA. However, after manual evaluation of the validation set, we find that by correcting gold label errors and improving dataset annotation schema, GPT-4 with our prompts can achieve (1) 89.6%-96.8% accuracy in DST, and (2) more than 99% correct response generation across different languages. This leads us to conclude that current automatic metrics heavily underestimate the effectiveness of in-context learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。