LLM对口语对话具有鲁棒性吗？
Are LLMs Robust for Spoken Dialogues?
AI总结：
本文在DSTC11测试集上评估了LLM在口语任务导向对话中的鲁棒性，发现默认模型对口语噪声不具鲁棒性，但在合适的口语对话数据集上微调后性能更鲁棒。
摘要：
大型预训练语言模型在包括对话状态跟踪和端到端响应生成在内的不同下游任务中展现了最先进的性能。然而，大多数公开可用的面向任务对话数据集和基准都集中在书面对话上。因此，所开发模型对口语交互的鲁棒性是未知的。在本研究中，我们在DSTC11测试集上评估了LLM在口语面向任务对话中的表现。由于缺乏合适的口语对话数据集，我们使用最先进的ASR引擎自动转录了一组口语对话开发集。我们表征了ASR错误类型及其分布，并在一个大型对话数据集中模拟了这些错误。我们分别报告了微调后的GPT-2和T5模型在响应生成和对话状态跟踪两个子任务中的内在（困惑度）和外在（人工评估）性能。结果表明，LLM默认对口语噪声不具鲁棒性，然而，在合适的口语TOD数据集上微调/训练此类模型可带来更鲁棒的性能。
英文摘要：
Large Pre-Trained Language Models have demonstrated state-of-the-art performance in different downstream tasks, including dialogue state tracking and end-to-end response generation. Nevertheless, most of the publicly available datasets and benchmarks on task-oriented dialogues focus on written conversations. Consequently, the robustness of the developed models to spoken interactions is unknown. In this work, we have evaluated the performance of LLMs for spoken task-oriented dialogues on the DSTC11 test sets. Due to the lack of proper spoken dialogue datasets, we have automatically transcribed a development set of spoken dialogues with a state-of-the-art ASR engine. We have characterized the ASR-error types and their distributions and simulated these errors in a large dataset of dialogues. We report the intrinsic (perplexity) and extrinsic (human evaluation) performance of fine-tuned GPT-2 and T5 models in two subtasks of response generation and dialogue state tracking, respectively. The results show that LLMs are not robust to spoken noise by default, however, fine-tuning/training such models on a proper dataset of spoken TODs can result in a more robust performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
