探索面向德语口语变体的任务型对话系统的鲁棒性
Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties
AI总结：
本研究针对德语口语变体，通过构建涵盖18种语言现象的扰动规则合成测试集，评估六种Transformer模型，发现任务型对话系统意图识别性能稳定，但槽位检测F1分数显著下降31%。
摘要：
主流跨语言任务型对话（ToD）系统利用迁移学习范式，在英语中训练一个用于意图识别和槽位填充的联合模型，并将其零样本应用于其他语言。我们解决了先前研究中的一个空白，该研究往往因测试数据有限而忽视向低资源口语变体的迁移。受先前关于英语变体研究的启发，我们设计并人工评估了将德语语句转换为口语形式的扰动规则，并使用这些规则在四个ToD数据集中合成测试集。我们的扰动规则涵盖18种不同的语言现象，使我们能够探索每种扰动对槽位和意图性能的影响。利用这些新数据集，我们对六种不同的Transformer模型进行了实验评估。在此，我们证明当应用于口语变体时，ToD系统能保持其意图识别性能，平均准确率仅下降6%（4.62个百分点）。然而，它们在槽位检测方面表现出显著下降，槽位F1分数下降了31%（21个百分点）。我们的发现进一步得到了从标准美国英语到合成城市非裔美国人白话英语的迁移实验的支持。
英文摘要：
Mainstream cross-lingual task-oriented dialogue (ToD) systems leverage the transfer learning paradigm by training a joint model for intent recognition and slot-filling in English and applying it, zero-shot, to other languages. We address a gap in prior research, which often overlooked the transfer to lower-resource colloquial varieties due to limited test data. Inspired by prior work on English varieties, we craft and manually evaluate perturbation rules that transform German sentences into colloquial forms and use them to synthesize test sets in four ToD datasets. Our perturbation rules cover 18 distinct language phenomena, enabling us to explore the impact of each perturbation on slot and intent performance. Using these new datasets, we conduct an experimental evaluation across six different transformers. Here, we demonstrate that when applied to colloquial varieties, ToD systems maintain their intent recognition performance, losing 6% (4.62 percentage points) in accuracy on average. However, they exhibit a significant drop in slot detection, with a decrease of 31% (21 percentage points) in slot F1 score. Our findings are further supported by a transfer experiment from Standard American English to synthetic Urban African American Vernacular English.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
